2026-02-22T21:43:46.810 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-22T21:43:46.823 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T21:43:46.902 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64109 branch: wip-yuri6-testing-2026-02-22-1629-squid description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '64109' kernel: branch: distro kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri6-testing-2026-02-22-1629-squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 workunit: branch: wip-yuri6-testing-2026-02-22-1629-squid sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5907 sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2026-02-22-1629-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 targets: trial034.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG2TDr2n3W6/SQ5S4PJW0RBXRBSxdN5BiEReN9tTpNPwdzrPxe6NjruOApTc6J7pKPXDI71l3TO9g9UyuPLkJWw= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-22_19:35:40 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-22T21:43:46.902 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa; will attempt to use it 2026-02-22T21:43:46.903 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f90f4eaebb28b619cf76e80cbbe997656da69579/qa/tasks 2026-02-22T21:43:46.903 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-22T21:43:46.904 INFO:teuthology.task.internal:Checking packages... 2026-02-22T21:43:46.904 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f90f4eaebb28b619cf76e80cbbe997656da69579' 2026-02-22T21:43:46.904 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-22T21:43:46.904 INFO:teuthology.packaging:ref: None 2026-02-22T21:43:46.904 INFO:teuthology.packaging:tag: None 2026-02-22T21:43:46.904 INFO:teuthology.packaging:branch: wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T21:43:46.904 INFO:teuthology.packaging:sha1: f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:43:46.905 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T21:43:47.021 INFO:teuthology.task.internal:Found packages for ceph version 19.2.3-689-gf90f4eae-1jammy 2026-02-22T21:43:47.023 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-22T21:43:47.034 INFO:teuthology.task.internal:no buildpackages task found 2026-02-22T21:43:47.034 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-22T21:43:47.048 INFO:teuthology.task.internal:Saving configuration 2026-02-22T21:43:47.055 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-22T21:43:47.066 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-22T21:43:47.147 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial034.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64109', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 21:40:17.188063', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC8NkFbr1mG4yMUOzJfxRBMywWJQo4X1VdzkEpHHSpAv'} 2026-02-22T21:43:47.147 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-22T21:43:47.158 INFO:teuthology.task.internal:roles: ubuntu@trial034.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-22T21:43:47.158 INFO:teuthology.run_tasks:Running task console_log... 2026-02-22T21:43:47.246 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x788fc1df6d40>, signals=[15]) 2026-02-22T21:43:47.246 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-22T21:43:47.256 INFO:teuthology.task.internal:Opening connections... 2026-02-22T21:43:47.256 DEBUG:teuthology.task.internal:connecting to ubuntu@trial034.front.sepia.ceph.com 2026-02-22T21:43:47.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial034.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T21:43:47.321 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-22T21:43:47.332 DEBUG:teuthology.orchestra.run.trial034:> uname -m 2026-02-22T21:43:47.399 INFO:teuthology.orchestra.run.trial034.stdout:x86_64 2026-02-22T21:43:47.400 DEBUG:teuthology.orchestra.run.trial034:> cat /etc/os-release 2026-02-22T21:43:47.446 INFO:teuthology.orchestra.run.trial034.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-22T21:43:47.446 INFO:teuthology.orchestra.run.trial034.stdout:NAME="Ubuntu" 2026-02-22T21:43:47.446 INFO:teuthology.orchestra.run.trial034.stdout:VERSION_ID="22.04" 2026-02-22T21:43:47.446 INFO:teuthology.orchestra.run.trial034.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-22T21:43:47.446 INFO:teuthology.orchestra.run.trial034.stdout:VERSION_CODENAME=jammy 2026-02-22T21:43:47.446 INFO:teuthology.orchestra.run.trial034.stdout:ID=ubuntu 2026-02-22T21:43:47.446 INFO:teuthology.orchestra.run.trial034.stdout:ID_LIKE=debian 2026-02-22T21:43:47.446 INFO:teuthology.orchestra.run.trial034.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-22T21:43:47.446 INFO:teuthology.orchestra.run.trial034.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-22T21:43:47.446 INFO:teuthology.orchestra.run.trial034.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-22T21:43:47.446 INFO:teuthology.orchestra.run.trial034.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-22T21:43:47.446 INFO:teuthology.orchestra.run.trial034.stdout:UBUNTU_CODENAME=jammy 2026-02-22T21:43:47.447 INFO:teuthology.lock.ops:Updating trial034.front.sepia.ceph.com on lock server 2026-02-22T21:43:47.520 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-22T21:43:47.531 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-22T21:43:47.542 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-22T21:43:47.542 DEBUG:teuthology.orchestra.run.trial034:> test '!' -e /home/ubuntu/cephtest 2026-02-22T21:43:47.546 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-22T21:43:47.556 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-22T21:43:47.556 DEBUG:teuthology.orchestra.run.trial034:> test -z $(ls -A /var/lib/ceph) 2026-02-22T21:43:47.590 INFO:teuthology.orchestra.run.trial034.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-22T21:43:47.591 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-22T21:43:47.685 INFO:teuthology.run_tasks:Running task kernel... 2026-02-22T21:43:47.703 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-22T21:43:47.703 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-22T21:43:47.703 DEBUG:teuthology.orchestra.run.trial034:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T21:43:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:43:47.707 DEBUG:teuthology.orchestra.run.trial034:> uname -r 2026-02-22T21:43:47.750 INFO:teuthology.orchestra.run.trial034.stdout:5.15.0-170-generic 2026-02-22T21:43:47.751 INFO:teuthology.task.kernel:Running kernel on trial034: 5.15.0-170-generic 2026-02-22T21:43:47.751 DEBUG:teuthology.orchestra.run.trial034:> sudo apt-get clean 2026-02-22T21:43:47.809 DEBUG:teuthology.orchestra.run.trial034:> sudo apt-get update 2026-02-22T21:43:48.248 INFO:teuthology.orchestra.run.trial034.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T21:43:48.321 INFO:teuthology.orchestra.run.trial034.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-22T21:43:48.595 INFO:teuthology.orchestra.run.trial034.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-22T21:43:48.716 INFO:teuthology.orchestra.run.trial034.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-22T21:43:48.830 INFO:teuthology.orchestra.run.trial034.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-22T21:43:49.158 INFO:teuthology.orchestra.run.trial034.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-22T21:43:49.167 INFO:teuthology.orchestra.run.trial034.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-22T21:43:49.168 INFO:teuthology.orchestra.run.trial034.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-22T21:43:49.196 INFO:teuthology.orchestra.run.trial034.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-22T21:43:49.202 INFO:teuthology.orchestra.run.trial034.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-22T21:43:49.202 INFO:teuthology.orchestra.run.trial034.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-22T21:43:49.330 INFO:teuthology.orchestra.run.trial034.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-22T21:43:49.349 INFO:teuthology.orchestra.run.trial034.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-22T21:43:49.420 INFO:teuthology.orchestra.run.trial034.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-22T21:43:49.427 INFO:teuthology.orchestra.run.trial034.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-22T21:43:49.428 INFO:teuthology.orchestra.run.trial034.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-22T21:43:49.456 INFO:teuthology.orchestra.run.trial034.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-22T21:43:49.456 INFO:teuthology.orchestra.run.trial034.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-22T21:43:49.578 INFO:teuthology.orchestra.run.trial034.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-22T21:43:53.096 INFO:teuthology.orchestra.run.trial034.stdout:Fetched 22.6 MB in 3s (8631 kB/s) 2026-02-22T21:43:53.615 INFO:teuthology.orchestra.run.trial034.stdout:Reading package lists... 2026-02-22T21:43:53.629 DEBUG:teuthology.orchestra.run.trial034:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-22T21:43:53.663 INFO:teuthology.orchestra.run.trial034.stdout:Reading package lists... 2026-02-22T21:43:53.769 INFO:teuthology.orchestra.run.trial034.stdout:Building dependency tree... 2026-02-22T21:43:53.770 INFO:teuthology.orchestra.run.trial034.stdout:Reading state information... 2026-02-22T21:43:53.854 INFO:teuthology.orchestra.run.trial034.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-22T21:43:53.854 INFO:teuthology.orchestra.run.trial034.stdout:linux-image-generic set to manually installed. 2026-02-22T21:43:53.854 INFO:teuthology.orchestra.run.trial034.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T21:43:53.855 INFO:teuthology.orchestra.run.trial034.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T21:43:53.855 INFO:teuthology.orchestra.run.trial034.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T21:43:53.855 INFO:teuthology.orchestra.run.trial034.stdout: sg3-utils-udev 2026-02-22T21:43:53.855 INFO:teuthology.orchestra.run.trial034.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T21:43:53.873 INFO:teuthology.orchestra.run.trial034.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T21:43:53.875 DEBUG:teuthology.orchestra.run.trial034:> dpkg -s linux-image-generic 2026-02-22T21:43:53.930 INFO:teuthology.orchestra.run.trial034.stdout:Package: linux-image-generic 2026-02-22T21:43:53.931 INFO:teuthology.orchestra.run.trial034.stdout:Status: install ok installed 2026-02-22T21:43:53.931 INFO:teuthology.orchestra.run.trial034.stdout:Priority: optional 2026-02-22T21:43:53.931 INFO:teuthology.orchestra.run.trial034.stdout:Section: kernel 2026-02-22T21:43:53.931 INFO:teuthology.orchestra.run.trial034.stdout:Installed-Size: 21 2026-02-22T21:43:53.931 INFO:teuthology.orchestra.run.trial034.stdout:Maintainer: Ubuntu Kernel Team 2026-02-22T21:43:53.931 INFO:teuthology.orchestra.run.trial034.stdout:Architecture: amd64 2026-02-22T21:43:53.932 INFO:teuthology.orchestra.run.trial034.stdout:Source: linux-meta 2026-02-22T21:43:53.932 INFO:teuthology.orchestra.run.trial034.stdout:Version: 5.15.0.170.159 2026-02-22T21:43:53.932 INFO:teuthology.orchestra.run.trial034.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-22T21:43:53.932 INFO:teuthology.orchestra.run.trial034.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-22T21:43:53.932 INFO:teuthology.orchestra.run.trial034.stdout:Recommends: thermald 2026-02-22T21:43:53.932 INFO:teuthology.orchestra.run.trial034.stdout:Description: Generic Linux kernel image 2026-02-22T21:43:53.933 INFO:teuthology.orchestra.run.trial034.stdout: This package will always depend on the latest generic kernel image 2026-02-22T21:43:53.933 INFO:teuthology.orchestra.run.trial034.stdout: available. 2026-02-22T21:43:53.933 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T21:43:53.934 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-22T21:43:53.934 DEBUG:teuthology.orchestra.run.trial034:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T21:43:53.979 INFO:teuthology.orchestra.run.trial034.stdout:ttyS1 2026-02-22T21:43:54.009 DEBUG:teuthology.parallel:result is None 2026-02-22T21:43:54.009 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-22T21:43:54.022 INFO:teuthology.task.internal:Creating test directory... 2026-02-22T21:43:54.022 DEBUG:teuthology.orchestra.run.trial034:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T21:43:54.054 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-22T21:43:54.066 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-22T21:43:54.079 INFO:teuthology.task.internal:Creating archive directory... 2026-02-22T21:43:54.079 DEBUG:teuthology.orchestra.run.trial034:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T21:43:54.106 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-22T21:43:54.118 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-22T21:43:54.118 DEBUG:teuthology.orchestra.run.trial034:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T21:43:54.151 INFO:teuthology.orchestra.run.trial034.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:43:54.157 INFO:teuthology.orchestra.run.trial034.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:43:54.158 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-22T21:43:54.170 INFO:teuthology.task.internal:Configuring sudo... 2026-02-22T21:43:54.170 DEBUG:teuthology.orchestra.run.trial034:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T21:43:54.207 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-22T21:43:54.224 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-22T21:43:54.225 DEBUG:teuthology.orchestra.run.trial034:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T21:43:54.254 DEBUG:teuthology.orchestra.run.trial034:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:43:54.298 DEBUG:teuthology.orchestra.run.trial034:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:43:54.343 DEBUG:teuthology.orchestra.run.trial034:> set -ex 2026-02-22T21:43:54.343 DEBUG:teuthology.orchestra.run.trial034:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T21:43:54.392 DEBUG:teuthology.orchestra.run.trial034:> sudo service rsyslog restart 2026-02-22T21:43:54.453 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-22T21:43:54.466 INFO:teuthology.task.internal:Starting timer... 2026-02-22T21:43:54.466 INFO:teuthology.run_tasks:Running task pcp... 2026-02-22T21:43:54.488 INFO:teuthology.run_tasks:Running task selinux... 2026-02-22T21:43:54.513 DEBUG:teuthology.task.selinux:Excluding trial034: OS 'ubuntu' does not support SELinux 2026-02-22T21:43:54.513 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-22T21:43:54.513 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-22T21:43:54.513 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-22T21:43:54.513 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-22T21:43:54.529 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2026-02-22T21:43:54.532 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2026-02-22T21:43:54.637 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-22T21:43:54.645 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-22T21:43:54.646 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial034.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-22T21:46:19.126 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial034.front.sepia.ceph.com')] 2026-02-22T21:46:19.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial034.front.sepia.ceph.com' 2026-02-22T21:46:19.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial034.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T21:46:19.238 DEBUG:teuthology.orchestra.run.trial034:> true 2026-02-22T21:46:19.488 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial034.front.sepia.ceph.com' 2026-02-22T21:46:19.489 INFO:teuthology.run_tasks:Running task clock... 2026-02-22T21:46:19.502 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-22T21:46:19.502 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T21:46:19.503 DEBUG:teuthology.orchestra.run.trial034:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T21:46:19.553 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-22T21:46:19.553 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: Command line: ntpd -gq 2026-02-22T21:46:19.554 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: ---------------------------------------------------- 2026-02-22T21:46:19.554 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: ntp-4 is maintained by Network Time Foundation, 2026-02-22T21:46:19.554 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-22T21:46:19.554 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: corporation. Support and training for ntp-4 are 2026-02-22T21:46:19.554 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: available at https://www.nwtime.org/support 2026-02-22T21:46:19.554 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: ---------------------------------------------------- 2026-02-22T21:46:19.554 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: proto: precision = 0.020 usec (-25) 2026-02-22T21:46:19.554 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: basedate set to 2022-02-04 2026-02-22T21:46:19.554 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: gps base set to 2022-02-06 (week 2196) 2026-02-22T21:46:19.554 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T21:46:19.555 INFO:teuthology.orchestra.run.trial034.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T21:46:19.555 INFO:teuthology.orchestra.run.trial034.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-22T21:46:19.556 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: restrict ::: KOD does nothing without LIMITED. 2026-02-22T21:46:19.557 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: Listen and drop on 0 v6wildcard [::]:123 2026-02-22T21:46:19.557 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-22T21:46:19.557 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: Listen normally on 2 lo 127.0.0.1:123 2026-02-22T21:46:19.557 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: Listen normally on 3 enp1s0f0 10.20.193.34:123 2026-02-22T21:46:19.557 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: Listen normally on 4 lo [::1]:123 2026-02-22T21:46:19.557 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:6032%2]:123 2026-02-22T21:46:19.557 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:19 ntpd[9440]: Listening on routing socket on fd #22 for interface updates 2026-02-22T21:46:20.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:20 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:20.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:20 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:20.642 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:20 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:20.642 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:20 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:20.642 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:20 ntpd[9440]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T21:46:20.642 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:20 ntpd[9440]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T21:46:21.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:21 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:21.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:21 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:21.556 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:21 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:21.556 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:21 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:21.556 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:21 ntpd[9440]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T21:46:21.557 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:21 ntpd[9440]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T21:46:22.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:22 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:22.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:22 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:22.641 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:22 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:22.641 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:22 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:22.641 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:22 ntpd[9440]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T21:46:22.642 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:22 ntpd[9440]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T21:46:23.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:23.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:23.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:23.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:23.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:23.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:23.556 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:23.556 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:23.556 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T21:46:23.556 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T21:46:23.621 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:23.621 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:23.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T21:46:23.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T21:46:23.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:23.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:23.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T21:46:23.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:23 ntpd[9440]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T21:46:24.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:24 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:24.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:24 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:24.641 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:24 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:24.641 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:24 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:24.642 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:24 ntpd[9440]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T21:46:24.642 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:24 ntpd[9440]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T21:46:25.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:25.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:25.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:25.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:25.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:25.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:25.556 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:25.556 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:25.556 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T21:46:25.556 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T21:46:25.621 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:25.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:25.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T21:46:25.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T21:46:25.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:25.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:25.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T21:46:25.622 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:25 ntpd[9440]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T21:46:26.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:26 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:26.553 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:26 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:26.641 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:26 ntpd[9440]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T21:46:26.642 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:26 ntpd[9440]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T21:46:26.642 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:26 ntpd[9440]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T21:46:26.642 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:26 ntpd[9440]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T21:46:26.643 INFO:teuthology.orchestra.run.trial034.stdout:22 Feb 21:46:26 ntpd[9440]: ntpd: time set -0.360959 s 2026-02-22T21:46:26.643 INFO:teuthology.orchestra.run.trial034.stdout:ntpd: time set -0.360959s 2026-02-22T21:46:26.644 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:26 ntpd[9440]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-22T21:46:26.644 INFO:teuthology.orchestra.run.trial034.stderr:22 Feb 21:46:26 ntpd[9440]: can't open /var/log/ntpstats/loopstats.20260222: Permission denied 2026-02-22T21:46:26.813 INFO:teuthology.orchestra.run.trial034.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T21:46:26.813 INFO:teuthology.orchestra.run.trial034.stdout:============================================================================== 2026-02-22T21:46:26.813 INFO:teuthology.orchestra.run.trial034.stdout: 204.2.134.173 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T21:46:26.813 INFO:teuthology.orchestra.run.trial034.stdout: 108.61.73.244 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T21:46:26.813 INFO:teuthology.orchestra.run.trial034.stdout: 23.186.168.132 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T21:46:26.813 INFO:teuthology.orchestra.run.trial034.stdout: 172-233-155-39. .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T21:46:26.814 INFO:teuthology.run_tasks:Running task install... 2026-02-22T21:46:26.829 DEBUG:teuthology.task.install:project ceph 2026-02-22T21:46:26.829 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'}} 2026-02-22T21:46:26.829 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'f90f4eaebb28b619cf76e80cbbe997656da69579'} 2026-02-22T21:46:26.829 INFO:teuthology.task.install:Using flavor: default 2026-02-22T21:46:26.832 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-22T21:46:26.833 INFO:teuthology.task.install:extra packages: [] 2026-02-22T21:46:26.833 DEBUG:teuthology.orchestra.run.trial034:> sudo apt-key list | grep Ceph 2026-02-22T21:46:26.914 INFO:teuthology.orchestra.run.trial034.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-22T21:46:26.944 INFO:teuthology.orchestra.run.trial034.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-22T21:46:26.944 INFO:teuthology.orchestra.run.trial034.stdout:uid [ unknown] Ceph.com (release key) 2026-02-22T21:46:26.945 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-22T21:46:26.945 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:46:27.118 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default/ 2026-02-22T21:46:27.118 INFO:teuthology.task.install.deb:Package version is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T21:46:27.251 DEBUG:teuthology.orchestra.run.trial034:> set -ex 2026-02-22T21:46:27.251 DEBUG:teuthology.orchestra.run.trial034:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-22T21:46:27.259 DEBUG:teuthology.orchestra.run.trial034:> sudo apt-get update 2026-02-22T21:46:27.412 INFO:teuthology.orchestra.run.trial034.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy InRelease 2026-02-22T21:46:27.423 INFO:teuthology.orchestra.run.trial034.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release [7662 B] 2026-02-22T21:46:27.437 INFO:teuthology.orchestra.run.trial034.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T21:46:27.437 INFO:teuthology.orchestra.run.trial034.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-22T21:46:27.441 INFO:teuthology.orchestra.run.trial034.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-22T21:46:27.448 INFO:teuthology.orchestra.run.trial034.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2026-02-22T21:46:27.450 INFO:teuthology.orchestra.run.trial034.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-22T21:46:27.458 INFO:teuthology.orchestra.run.trial034.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-22T21:46:27.777 INFO:teuthology.orchestra.run.trial034.stdout:Fetched 25.8 kB in 0s (72.5 kB/s) 2026-02-22T21:46:28.289 INFO:teuthology.orchestra.run.trial034.stdout:Reading package lists... 2026-02-22T21:46:28.302 DEBUG:teuthology.orchestra.run.trial034:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.2.3-689-gf90f4eae-1jammy cephadm=19.2.3-689-gf90f4eae-1jammy ceph-mds=19.2.3-689-gf90f4eae-1jammy ceph-mgr=19.2.3-689-gf90f4eae-1jammy ceph-common=19.2.3-689-gf90f4eae-1jammy ceph-fuse=19.2.3-689-gf90f4eae-1jammy ceph-test=19.2.3-689-gf90f4eae-1jammy ceph-volume=19.2.3-689-gf90f4eae-1jammy radosgw=19.2.3-689-gf90f4eae-1jammy python3-rados=19.2.3-689-gf90f4eae-1jammy python3-rgw=19.2.3-689-gf90f4eae-1jammy python3-cephfs=19.2.3-689-gf90f4eae-1jammy python3-rbd=19.2.3-689-gf90f4eae-1jammy libcephfs2=19.2.3-689-gf90f4eae-1jammy libcephfs-dev=19.2.3-689-gf90f4eae-1jammy librados2=19.2.3-689-gf90f4eae-1jammy librbd1=19.2.3-689-gf90f4eae-1jammy rbd-fuse=19.2.3-689-gf90f4eae-1jammy 2026-02-22T21:46:28.335 INFO:teuthology.orchestra.run.trial034.stdout:Reading package lists... 2026-02-22T21:46:28.441 INFO:teuthology.orchestra.run.trial034.stdout:Building dependency tree... 2026-02-22T21:46:28.442 INFO:teuthology.orchestra.run.trial034.stdout:Reading state information... 2026-02-22T21:46:28.532 INFO:teuthology.orchestra.run.trial034.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T21:46:28.532 INFO:teuthology.orchestra.run.trial034.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T21:46:28.533 INFO:teuthology.orchestra.run.trial034.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T21:46:28.533 INFO:teuthology.orchestra.run.trial034.stdout: sg3-utils-udev 2026-02-22T21:46:28.533 INFO:teuthology.orchestra.run.trial034.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T21:46:28.533 INFO:teuthology.orchestra.run.trial034.stdout:The following additional packages will be installed: 2026-02-22T21:46:28.533 INFO:teuthology.orchestra.run.trial034.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-22T21:46:28.533 INFO:teuthology.orchestra.run.trial034.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-22T21:46:28.533 INFO:teuthology.orchestra.run.trial034.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 2026-02-22T21:46:28.533 INFO:teuthology.orchestra.run.trial034.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-22T21:46:28.533 INFO:teuthology.orchestra.run.trial034.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2026-02-22T21:46:28.533 INFO:teuthology.orchestra.run.trial034.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2026-02-22T21:46:28.533 INFO:teuthology.orchestra.run.trial034.stdout: pkg-config python-asyncssh-doc python-pastedeploy-tpl python3-asyncssh 2026-02-22T21:46:28.534 INFO:teuthology.orchestra.run.trial034.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cheroot 2026-02-22T21:46:28.534 INFO:teuthology.orchestra.run.trial034.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T21:46:28.534 INFO:teuthology.orchestra.run.trial034.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T21:46:28.534 INFO:teuthology.orchestra.run.trial034.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T21:46:28.534 INFO:teuthology.orchestra.run.trial034.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T21:46:28.534 INFO:teuthology.orchestra.run.trial034.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T21:46:28.534 INFO:teuthology.orchestra.run.trial034.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T21:46:28.534 INFO:teuthology.orchestra.run.trial034.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2026-02-22T21:46:28.534 INFO:teuthology.orchestra.run.trial034.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2026-02-22T21:46:28.534 INFO:teuthology.orchestra.run.trial034.stdout: python3-simplejson python3-singledispatch python3-sklearn 2026-02-22T21:46:28.534 INFO:teuthology.orchestra.run.trial034.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2026-02-22T21:46:28.534 INFO:teuthology.orchestra.run.trial034.stdout: python3-toml python3-waitress python3-wcwidth python3-webob 2026-02-22T21:46:28.535 INFO:teuthology.orchestra.run.trial034.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2026-02-22T21:46:28.535 INFO:teuthology.orchestra.run.trial034.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2026-02-22T21:46:28.535 INFO:teuthology.orchestra.run.trial034.stdout:Suggested packages: 2026-02-22T21:46:28.535 INFO:teuthology.orchestra.run.trial034.stdout: python3-influxdb readline-doc python3-beaker python-mako-doc 2026-02-22T21:46:28.535 INFO:teuthology.orchestra.run.trial034.stdout: python-natsort-doc httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2026-02-22T21:46:28.535 INFO:teuthology.orchestra.run.trial034.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2026-02-22T21:46:28.535 INFO:teuthology.orchestra.run.trial034.stdout: python-pygments-doc ttf-bitstream-vera python-pyinotify-doc python3-dap 2026-02-22T21:46:28.535 INFO:teuthology.orchestra.run.trial034.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2026-02-22T21:46:28.535 INFO:teuthology.orchestra.run.trial034.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2026-02-22T21:46:28.535 INFO:teuthology.orchestra.run.trial034.stdout:Recommended packages: 2026-02-22T21:46:28.535 INFO:teuthology.orchestra.run.trial034.stdout: btrfs-tools 2026-02-22T21:46:28.566 INFO:teuthology.orchestra.run.trial034.stdout:The following NEW packages will be installed: 2026-02-22T21:46:28.566 INFO:teuthology.orchestra.run.trial034.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-22T21:46:28.566 INFO:teuthology.orchestra.run.trial034.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-22T21:46:28.566 INFO:teuthology.orchestra.run.trial034.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-22T21:46:28.566 INFO:teuthology.orchestra.run.trial034.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 2026-02-22T21:46:28.566 INFO:teuthology.orchestra.run.trial034.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-22T21:46:28.566 INFO:teuthology.orchestra.run.trial034.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2026-02-22T21:46:28.566 INFO:teuthology.orchestra.run.trial034.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2026-02-22T21:46:28.566 INFO:teuthology.orchestra.run.trial034.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2026-02-22T21:46:28.566 INFO:teuthology.orchestra.run.trial034.stdout: python-pastedeploy-tpl python3-asyncssh python3-cachetools 2026-02-22T21:46:28.566 INFO:teuthology.orchestra.run.trial034.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cheroot 2026-02-22T21:46:28.566 INFO:teuthology.orchestra.run.trial034.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: python3-threadpoolctl python3-toml python3-waitress python3-wcwidth 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse socat unzip 2026-02-22T21:46:28.567 INFO:teuthology.orchestra.run.trial034.stdout: xmlstarlet zip 2026-02-22T21:46:28.647 INFO:teuthology.orchestra.run.trial034.stdout:0 upgraded, 108 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T21:46:28.647 INFO:teuthology.orchestra.run.trial034.stdout:Need to get 178 MB of archives. 2026-02-22T21:46:28.647 INFO:teuthology.orchestra.run.trial034.stdout:After this operation, 809 MB of additional disk space will be used. 2026-02-22T21:46:28.647 INFO:teuthology.orchestra.run.trial034.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.2.3-689-gf90f4eae-1jammy [3597 kB] 2026-02-22T21:46:28.749 INFO:teuthology.orchestra.run.trial034.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.2.3-689-gf90f4eae-1jammy [3256 kB] 2026-02-22T21:46:28.794 INFO:teuthology.orchestra.run.trial034.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.2.3-689-gf90f4eae-1jammy [979 kB] 2026-02-22T21:46:28.810 INFO:teuthology.orchestra.run.trial034.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.2.3-689-gf90f4eae-1jammy [357 kB] 2026-02-22T21:46:28.815 INFO:teuthology.orchestra.run.trial034.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.2.3-689-gf90f4eae-1jammy [32.9 kB] 2026-02-22T21:46:28.816 INFO:teuthology.orchestra.run.trial034.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.2.3-689-gf90f4eae-1jammy [184 kB] 2026-02-22T21:46:28.819 INFO:teuthology.orchestra.run.trial034.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.2.3-689-gf90f4eae-1jammy [70.0 kB] 2026-02-22T21:46:28.820 INFO:teuthology.orchestra.run.trial034.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.2.3-689-gf90f4eae-1jammy [335 kB] 2026-02-22T21:46:28.825 INFO:teuthology.orchestra.run.trial034.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.2.3-689-gf90f4eae-1jammy [6935 kB] 2026-02-22T21:46:28.921 INFO:teuthology.orchestra.run.trial034.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-22T21:46:28.940 INFO:teuthology.orchestra.run.trial034.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.2.3-689-gf90f4eae-1jammy [112 kB] 2026-02-22T21:46:28.941 INFO:teuthology.orchestra.run.trial034.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.2.3-689-gf90f4eae-1jammy [470 kB] 2026-02-22T21:46:28.948 INFO:teuthology.orchestra.run.trial034.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.2.3-689-gf90f4eae-1jammy [26.5 MB] 2026-02-22T21:46:29.257 INFO:teuthology.orchestra.run.trial034.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-22T21:46:29.267 INFO:teuthology.orchestra.run.trial034.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-22T21:46:29.337 INFO:teuthology.orchestra.run.trial034.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-22T21:46:29.389 INFO:teuthology.orchestra.run.trial034.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.2.3-689-gf90f4eae-1jammy [5178 kB] 2026-02-22T21:46:29.473 INFO:teuthology.orchestra.run.trial034.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.2.3-689-gf90f4eae-1jammy [248 kB] 2026-02-22T21:46:29.475 INFO:teuthology.orchestra.run.trial034.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.2.3-689-gf90f4eae-1jammy [125 kB] 2026-02-22T21:46:29.477 INFO:teuthology.orchestra.run.trial034.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.2.3-689-gf90f4eae-1jammy [1081 kB] 2026-02-22T21:46:29.496 INFO:teuthology.orchestra.run.trial034.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.2.3-689-gf90f4eae-1jammy [6239 kB] 2026-02-22T21:46:29.538 INFO:teuthology.orchestra.run.trial034.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-22T21:46:29.548 INFO:teuthology.orchestra.run.trial034.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-22T21:46:29.575 INFO:teuthology.orchestra.run.trial034.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-22T21:46:29.583 INFO:teuthology.orchestra.run.trial034.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-22T21:46:29.585 INFO:teuthology.orchestra.run.trial034.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-22T21:46:29.586 INFO:teuthology.orchestra.run.trial034.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-22T21:46:29.587 INFO:teuthology.orchestra.run.trial034.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-22T21:46:29.601 INFO:teuthology.orchestra.run.trial034.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.2.3-689-gf90f4eae-1jammy [23.0 MB] 2026-02-22T21:46:29.603 INFO:teuthology.orchestra.run.trial034.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2026-02-22T21:46:29.607 INFO:teuthology.orchestra.run.trial034.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2026-02-22T21:46:29.611 INFO:teuthology.orchestra.run.trial034.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2026-02-22T21:46:29.679 INFO:teuthology.orchestra.run.trial034.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5034 B] 2026-02-22T21:46:29.680 INFO:teuthology.orchestra.run.trial034.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2026-02-22T21:46:29.681 INFO:teuthology.orchestra.run.trial034.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2026-02-22T21:46:29.683 INFO:teuthology.orchestra.run.trial034.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2026-02-22T21:46:29.685 INFO:teuthology.orchestra.run.trial034.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-22T21:46:29.685 INFO:teuthology.orchestra.run.trial034.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-22T21:46:29.686 INFO:teuthology.orchestra.run.trial034.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-22T21:46:29.686 INFO:teuthology.orchestra.run.trial034.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-22T21:46:29.757 INFO:teuthology.orchestra.run.trial034.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-22T21:46:29.757 INFO:teuthology.orchestra.run.trial034.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-22T21:46:29.758 INFO:teuthology.orchestra.run.trial034.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-22T21:46:29.758 INFO:teuthology.orchestra.run.trial034.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-22T21:46:29.758 INFO:teuthology.orchestra.run.trial034.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-22T21:46:29.828 INFO:teuthology.orchestra.run.trial034.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-22T21:46:29.830 INFO:teuthology.orchestra.run.trial034.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-22T21:46:29.831 INFO:teuthology.orchestra.run.trial034.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2026-02-22T21:46:29.831 INFO:teuthology.orchestra.run.trial034.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2026-02-22T21:46:29.832 INFO:teuthology.orchestra.run.trial034.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2026-02-22T21:46:29.900 INFO:teuthology.orchestra.run.trial034.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7320 B] 2026-02-22T21:46:29.901 INFO:teuthology.orchestra.run.trial034.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-22T21:46:29.901 INFO:teuthology.orchestra.run.trial034.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-waitress all 1.4.4-1.1ubuntu1.1 [47.0 kB] 2026-02-22T21:46:29.902 INFO:teuthology.orchestra.run.trial034.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2026-02-22T21:46:29.902 INFO:teuthology.orchestra.run.trial034.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2026-02-22T21:46:29.972 INFO:teuthology.orchestra.run.trial034.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4892 B] 2026-02-22T21:46:29.973 INFO:teuthology.orchestra.run.trial034.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2026-02-22T21:46:29.973 INFO:teuthology.orchestra.run.trial034.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2026-02-22T21:46:29.973 INFO:teuthology.orchestra.run.trial034.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2026-02-22T21:46:29.974 INFO:teuthology.orchestra.run.trial034.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.3 [181 kB] 2026-02-22T21:46:29.991 INFO:teuthology.orchestra.run.trial034.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.2.3-689-gf90f4eae-1jammy [14.2 kB] 2026-02-22T21:46:29.992 INFO:teuthology.orchestra.run.trial034.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.2.3-689-gf90f4eae-1jammy [1173 kB] 2026-02-22T21:46:30.008 INFO:teuthology.orchestra.run.trial034.stdout:Get:63 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.2.3-689-gf90f4eae-1jammy [2503 kB] 2026-02-22T21:46:30.044 INFO:teuthology.orchestra.run.trial034.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-22T21:46:30.045 INFO:teuthology.orchestra.run.trial034.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-22T21:46:30.047 INFO:teuthology.orchestra.run.trial034.stdout:Get:66 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-22T21:46:30.048 INFO:teuthology.orchestra.run.trial034.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-22T21:46:30.048 INFO:teuthology.orchestra.run.trial034.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-22T21:46:30.050 INFO:teuthology.orchestra.run.trial034.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.2.3-689-gf90f4eae-1jammy [798 kB] 2026-02-22T21:46:30.065 INFO:teuthology.orchestra.run.trial034.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.2.3-689-gf90f4eae-1jammy [157 kB] 2026-02-22T21:46:30.067 INFO:teuthology.orchestra.run.trial034.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.2.3-689-gf90f4eae-1jammy [2396 kB] 2026-02-22T21:46:30.107 INFO:teuthology.orchestra.run.trial034.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.2.3-689-gf90f4eae-1jammy [8625 kB] 2026-02-22T21:46:30.116 INFO:teuthology.orchestra.run.trial034.stdout:Get:73 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-22T21:46:30.118 INFO:teuthology.orchestra.run.trial034.stdout:Get:74 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-22T21:46:30.118 INFO:teuthology.orchestra.run.trial034.stdout:Get:75 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-22T21:46:30.139 INFO:teuthology.orchestra.run.trial034.stdout:Get:76 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-22T21:46:30.139 INFO:teuthology.orchestra.run.trial034.stdout:Get:77 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-22T21:46:30.188 INFO:teuthology.orchestra.run.trial034.stdout:Get:78 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-22T21:46:30.189 INFO:teuthology.orchestra.run.trial034.stdout:Get:79 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-22T21:46:30.189 INFO:teuthology.orchestra.run.trial034.stdout:Get:80 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-22T21:46:30.189 INFO:teuthology.orchestra.run.trial034.stdout:Get:81 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-22T21:46:30.192 INFO:teuthology.orchestra.run.trial034.stdout:Get:82 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-22T21:46:30.256 INFO:teuthology.orchestra.run.trial034.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.2.3-689-gf90f4eae-1jammy [14.3 kB] 2026-02-22T21:46:30.256 INFO:teuthology.orchestra.run.trial034.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.2.3-689-gf90f4eae-1jammy [52.1 MB] 2026-02-22T21:46:30.260 INFO:teuthology.orchestra.run.trial034.stdout:Get:85 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-22T21:46:30.261 INFO:teuthology.orchestra.run.trial034.stdout:Get:86 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-22T21:46:30.261 INFO:teuthology.orchestra.run.trial034.stdout:Get:87 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-22T21:46:30.265 INFO:teuthology.orchestra.run.trial034.stdout:Get:88 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-22T21:46:30.267 INFO:teuthology.orchestra.run.trial034.stdout:Get:89 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2026-02-22T21:46:30.332 INFO:teuthology.orchestra.run.trial034.stdout:Get:90 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2026-02-22T21:46:30.332 INFO:teuthology.orchestra.run.trial034.stdout:Get:91 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-22T21:46:30.337 INFO:teuthology.orchestra.run.trial034.stdout:Get:92 https://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2026-02-22T21:46:30.337 INFO:teuthology.orchestra.run.trial034.stdout:Get:93 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-22T21:46:30.340 INFO:teuthology.orchestra.run.trial034.stdout:Get:94 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-22T21:46:30.403 INFO:teuthology.orchestra.run.trial034.stdout:Get:95 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2026-02-22T21:46:30.404 INFO:teuthology.orchestra.run.trial034.stdout:Get:96 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-22T21:46:30.404 INFO:teuthology.orchestra.run.trial034.stdout:Get:97 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-22T21:46:30.406 INFO:teuthology.orchestra.run.trial034.stdout:Get:98 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-22T21:46:30.406 INFO:teuthology.orchestra.run.trial034.stdout:Get:99 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-22T21:46:30.476 INFO:teuthology.orchestra.run.trial034.stdout:Get:100 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2026-02-22T21:46:30.476 INFO:teuthology.orchestra.run.trial034.stdout:Get:101 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-22T21:46:30.476 INFO:teuthology.orchestra.run.trial034.stdout:Get:102 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-22T21:46:30.478 INFO:teuthology.orchestra.run.trial034.stdout:Get:103 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-22T21:46:30.478 INFO:teuthology.orchestra.run.trial034.stdout:Get:104 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-22T21:46:31.175 INFO:teuthology.orchestra.run.trial034.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.2.3-689-gf90f4eae-1jammy [135 kB] 2026-02-22T21:46:31.176 INFO:teuthology.orchestra.run.trial034.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.2.3-689-gf90f4eae-1jammy [41.0 kB] 2026-02-22T21:46:31.176 INFO:teuthology.orchestra.run.trial034.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.2.3-689-gf90f4eae-1jammy [13.7 MB] 2026-02-22T21:46:31.424 INFO:teuthology.orchestra.run.trial034.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2026-02-22-1629-squid/f90f4eaebb28b619cf76e80cbbe997656da69579/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.2.3-689-gf90f4eae-1jammy [92.3 kB] 2026-02-22T21:46:31.739 INFO:teuthology.orchestra.run.trial034.stdout:Fetched 178 MB in 3s (62.2 MB/s) 2026-02-22T21:46:31.756 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-22T21:46:31.919 INFO:teuthology.orchestra.run.trial034.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-22T21:46:31.921 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-22T21:46:31.923 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T21:46:31.944 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-22T21:46:31.956 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-22T21:46:31.956 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T21:46:31.972 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-22T21:46:31.984 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-22T21:46:31.984 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T21:46:32.004 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-22T21:46:32.012 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T21:46:32.017 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T21:46:32.046 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-22T21:46:32.054 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T21:46:32.055 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T21:46:32.071 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-22T21:46:32.079 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T21:46:32.079 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T21:46:32.099 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-22T21:46:32.107 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-22T21:46:32.108 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T21:46:32.124 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package librados2. 2026-02-22T21:46:32.132 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../007-librados2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:32.133 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:32.175 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libnbd0. 2026-02-22T21:46:32.183 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2026-02-22T21:46:32.184 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-22T21:46:32.199 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package librbd1. 2026-02-22T21:46:32.207 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../009-librbd1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:32.207 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:32.250 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libcephfs2. 2026-02-22T21:46:32.258 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../010-libcephfs2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:32.259 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:32.280 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-rados. 2026-02-22T21:46:32.288 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../011-python3-rados_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:32.288 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:32.304 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-22T21:46:32.313 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T21:46:32.313 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:32.327 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-cephfs. 2026-02-22T21:46:32.336 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../013-python3-cephfs_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:32.336 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:32.351 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-22T21:46:32.359 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../014-python3-ceph-common_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T21:46:32.360 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:32.377 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-22T21:46:32.385 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-22T21:46:32.385 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T21:46:32.400 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-prettytable. 2026-02-22T21:46:32.408 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2026-02-22T21:46:32.409 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-22T21:46:32.423 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-rbd. 2026-02-22T21:46:32.431 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../017-python3-rbd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:32.432 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:32.448 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-22T21:46:32.456 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-22T21:46:32.457 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T21:46:32.475 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libreadline-dev:amd64. 2026-02-22T21:46:32.483 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2026-02-22T21:46:32.484 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T21:46:32.500 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2026-02-22T21:46:32.509 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2026-02-22T21:46:32.509 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T21:46:32.527 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package lua5.1. 2026-02-22T21:46:32.535 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2026-02-22T21:46:32.535 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2026-02-22T21:46:32.557 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package lua-any. 2026-02-22T21:46:32.564 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2026-02-22T21:46:32.565 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking lua-any (27ubuntu1) ... 2026-02-22T21:46:32.579 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package zip. 2026-02-22T21:46:32.587 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2026-02-22T21:46:32.587 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking zip (3.0-12build2) ... 2026-02-22T21:46:32.603 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package unzip. 2026-02-22T21:46:32.612 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2026-02-22T21:46:32.612 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2026-02-22T21:46:32.629 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package luarocks. 2026-02-22T21:46:32.637 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2026-02-22T21:46:32.638 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2026-02-22T21:46:32.676 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package librgw2. 2026-02-22T21:46:32.684 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../026-librgw2_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:32.684 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:32.758 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-rgw. 2026-02-22T21:46:32.766 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../027-python3-rgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:32.767 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:32.782 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-22T21:46:32.790 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../028-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-22T21:46:32.790 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T21:46:32.805 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libradosstriper1. 2026-02-22T21:46:32.814 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../029-libradosstriper1_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:32.814 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:32.832 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-common. 2026-02-22T21:46:32.840 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../030-ceph-common_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:32.841 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:33.045 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-base. 2026-02-22T21:46:33.053 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../031-ceph-base_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:33.058 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:33.124 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-22T21:46:33.132 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-22T21:46:33.133 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-22T21:46:33.147 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-cheroot. 2026-02-22T21:46:33.155 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-22T21:46:33.156 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T21:46:33.173 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-22T21:46:33.181 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-22T21:46:33.182 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-22T21:46:33.196 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-22T21:46:33.204 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-22T21:46:33.205 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-22T21:46:33.219 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-22T21:46:33.228 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-22T21:46:33.228 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-22T21:46:33.242 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-tempora. 2026-02-22T21:46:33.250 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2026-02-22T21:46:33.251 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-22T21:46:33.266 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-portend. 2026-02-22T21:46:33.274 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2026-02-22T21:46:33.275 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-22T21:46:33.288 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-22T21:46:33.295 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-22T21:46:33.296 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-22T21:46:33.309 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-22T21:46:33.318 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-22T21:46:33.318 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-22T21:46:33.340 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-natsort. 2026-02-22T21:46:33.349 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2026-02-22T21:46:33.349 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-22T21:46:33.364 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-logutils. 2026-02-22T21:46:33.372 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2026-02-22T21:46:33.372 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-logutils (0.3.3-8) ... 2026-02-22T21:46:33.387 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-mako. 2026-02-22T21:46:33.395 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2026-02-22T21:46:33.396 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T21:46:33.413 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-simplegeneric. 2026-02-22T21:46:33.421 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2026-02-22T21:46:33.421 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2026-02-22T21:46:33.435 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-singledispatch. 2026-02-22T21:46:33.443 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2026-02-22T21:46:33.444 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2026-02-22T21:46:33.457 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-webob. 2026-02-22T21:46:33.465 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-22T21:46:33.465 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T21:46:33.481 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-waitress. 2026-02-22T21:46:33.490 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1.1_all.deb ... 2026-02-22T21:46:33.492 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T21:46:33.508 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-tempita. 2026-02-22T21:46:33.516 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2026-02-22T21:46:33.517 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T21:46:33.531 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-paste. 2026-02-22T21:46:33.539 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2026-02-22T21:46:33.539 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T21:46:33.567 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2026-02-22T21:46:33.575 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2026-02-22T21:46:33.575 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T21:46:33.589 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-pastedeploy. 2026-02-22T21:46:33.597 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2026-02-22T21:46:33.597 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2026-02-22T21:46:33.613 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-webtest. 2026-02-22T21:46:33.622 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2026-02-22T21:46:33.622 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-webtest (2.0.35-1) ... 2026-02-22T21:46:33.637 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-pecan. 2026-02-22T21:46:33.645 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2026-02-22T21:46:33.646 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T21:46:33.670 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-werkzeug. 2026-02-22T21:46:33.678 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.3_all.deb ... 2026-02-22T21:46:33.678 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T21:46:33.699 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-22T21:46:33.707 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T21:46:33.708 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:33.736 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-22T21:46:33.744 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:33.745 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:33.759 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-mgr. 2026-02-22T21:46:33.767 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../057-ceph-mgr_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:33.768 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:33.793 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-mon. 2026-02-22T21:46:33.801 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../058-ceph-mon_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:33.802 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:33.857 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-22T21:46:33.866 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../059-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-22T21:46:33.866 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T21:46:33.883 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-osd. 2026-02-22T21:46:33.890 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../060-ceph-osd_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:33.891 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:34.044 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph. 2026-02-22T21:46:34.054 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../061-ceph_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:34.054 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:34.068 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-fuse. 2026-02-22T21:46:34.076 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../062-ceph-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:34.076 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:34.099 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-mds. 2026-02-22T21:46:34.108 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../063-ceph-mds_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:34.108 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:34.142 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package cephadm. 2026-02-22T21:46:34.150 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../064-cephadm_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:34.151 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:34.167 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-22T21:46:34.176 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T21:46:34.176 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T21:46:34.198 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-22T21:46:34.207 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../066-ceph-mgr-cephadm_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T21:46:34.207 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:34.227 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-22T21:46:34.236 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../067-python3-repoze.lru_0.7-2_all.deb ... 2026-02-22T21:46:34.236 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-22T21:46:34.252 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-routes. 2026-02-22T21:46:34.260 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../068-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-22T21:46:34.261 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T21:46:34.281 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-22T21:46:34.290 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../069-ceph-mgr-dashboard_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T21:46:34.290 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:34.491 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-22T21:46:34.500 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../070-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-22T21:46:34.500 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T21:46:34.539 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-joblib. 2026-02-22T21:46:34.548 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../071-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-22T21:46:34.548 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T21:46:34.575 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-22T21:46:34.584 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../072-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-22T21:46:34.584 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-22T21:46:34.600 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-sklearn. 2026-02-22T21:46:34.608 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../073-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-22T21:46:34.609 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T21:46:34.689 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-22T21:46:34.698 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../074-ceph-mgr-diskprediction-local_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T21:46:34.698 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:34.866 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-cachetools. 2026-02-22T21:46:34.874 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../075-python3-cachetools_5.0.0-1_all.deb ... 2026-02-22T21:46:34.875 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-22T21:46:34.890 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-rsa. 2026-02-22T21:46:34.898 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../076-python3-rsa_4.8-1_all.deb ... 2026-02-22T21:46:34.899 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-22T21:46:34.916 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-google-auth. 2026-02-22T21:46:34.924 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-3_all.deb ... 2026-02-22T21:46:34.925 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-22T21:46:34.943 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-22T21:46:34.952 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-22T21:46:34.952 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T21:46:34.968 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-websocket. 2026-02-22T21:46:34.976 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../079-python3-websocket_1.2.3-1_all.deb ... 2026-02-22T21:46:34.977 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-22T21:46:34.994 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-22T21:46:35.003 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../080-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-22T21:46:35.003 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T21:46:35.099 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-22T21:46:35.108 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T21:46:35.108 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:35.123 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-22T21:46:35.131 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../082-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-22T21:46:35.138 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T21:46:35.155 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-22T21:46:35.163 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../083-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T21:46:35.164 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T21:46:35.180 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package jq. 2026-02-22T21:46:35.188 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../084-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T21:46:35.189 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-22T21:46:35.203 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package socat. 2026-02-22T21:46:35.211 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../085-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-22T21:46:35.212 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-22T21:46:35.232 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package xmlstarlet. 2026-02-22T21:46:35.240 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-22T21:46:35.240 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-22T21:46:35.278 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-test. 2026-02-22T21:46:35.286 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../087-ceph-test_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:35.286 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:35.746 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package ceph-volume. 2026-02-22T21:46:35.755 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../088-ceph-volume_19.2.3-689-gf90f4eae-1jammy_all.deb ... 2026-02-22T21:46:35.755 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:35.779 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-22T21:46:35.787 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../089-libcephfs-dev_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:35.788 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:35.803 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package lua-socket:amd64. 2026-02-22T21:46:35.812 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../090-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2026-02-22T21:46:35.813 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T21:46:35.834 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package lua-sec:amd64. 2026-02-22T21:46:35.842 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../091-lua-sec_1.0.2-1_amd64.deb ... 2026-02-22T21:46:35.843 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2026-02-22T21:46:35.860 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package nvme-cli. 2026-02-22T21:46:35.869 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../092-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-22T21:46:35.870 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T21:46:35.901 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package pkg-config. 2026-02-22T21:46:35.909 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../093-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2026-02-22T21:46:35.909 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T21:46:35.925 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-22T21:46:35.934 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../094-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T21:46:35.934 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T21:46:35.964 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-22T21:46:35.972 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../095-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-22T21:46:35.973 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-22T21:46:35.988 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-pastescript. 2026-02-22T21:46:35.997 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../096-python3-pastescript_2.0.2-4_all.deb ... 2026-02-22T21:46:35.997 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2026-02-22T21:46:36.018 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-pluggy. 2026-02-22T21:46:36.026 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../097-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-22T21:46:36.027 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-22T21:46:36.042 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-psutil. 2026-02-22T21:46:36.050 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../098-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-22T21:46:36.051 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-22T21:46:36.070 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-py. 2026-02-22T21:46:36.078 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../099-python3-py_1.10.0-1_all.deb ... 2026-02-22T21:46:36.079 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-22T21:46:36.098 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-pygments. 2026-02-22T21:46:36.106 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../100-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-22T21:46:36.107 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T21:46:36.148 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-pyinotify. 2026-02-22T21:46:36.158 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../101-python3-pyinotify_0.9.6-1.3_all.deb ... 2026-02-22T21:46:36.158 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2026-02-22T21:46:36.173 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-toml. 2026-02-22T21:46:36.181 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../102-python3-toml_0.10.2-1_all.deb ... 2026-02-22T21:46:36.181 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-22T21:46:36.197 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-pytest. 2026-02-22T21:46:36.205 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../103-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-22T21:46:36.206 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T21:46:36.228 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package python3-simplejson. 2026-02-22T21:46:36.236 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../104-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-22T21:46:36.236 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-22T21:46:36.255 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-22T21:46:36.263 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../105-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-22T21:46:36.264 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-22T21:46:36.332 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package radosgw. 2026-02-22T21:46:36.341 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../106-radosgw_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:36.342 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:36.455 INFO:teuthology.orchestra.run.trial034.stdout:Selecting previously unselected package rbd-fuse. 2026-02-22T21:46:36.464 INFO:teuthology.orchestra.run.trial034.stdout:Preparing to unpack .../107-rbd-fuse_19.2.3-689-gf90f4eae-1jammy_amd64.deb ... 2026-02-22T21:46:36.464 INFO:teuthology.orchestra.run.trial034.stdout:Unpacking rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:36.493 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-22T21:46:36.560 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T21:46:36.561 INFO:teuthology.orchestra.run.trial034.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T21:46:36.632 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T21:46:36.913 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-22T21:46:37.290 INFO:teuthology.orchestra.run.trial034.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-22T21:46:37.302 INFO:teuthology.orchestra.run.trial034.stdout:Setting up cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:37.345 INFO:teuthology.orchestra.run.trial034.stdout:Adding system user cephadm....done 2026-02-22T21:46:37.354 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-22T21:46:37.426 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-22T21:46:37.498 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T21:46:37.499 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-22T21:46:37.572 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-22T21:46:37.650 INFO:teuthology.orchestra.run.trial034.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T21:46:37.651 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-22T21:46:37.745 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T21:46:37.865 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-22T21:46:37.942 INFO:teuthology.orchestra.run.trial034.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2026-02-22T21:46:37.951 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2026-02-22T21:46:38.028 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-22T21:46:38.101 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-ceph-argparse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:38.170 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T21:46:38.171 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-22T21:46:38.172 INFO:teuthology.orchestra.run.trial034.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-22T21:46:38.173 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2026-02-22T21:46:38.173 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T21:46:38.174 INFO:teuthology.orchestra.run.trial034.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2026-02-22T21:46:38.191 INFO:teuthology.orchestra.run.trial034.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2026-02-22T21:46:38.193 INFO:teuthology.orchestra.run.trial034.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2026-02-22T21:46:38.194 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T21:46:38.195 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-22T21:46:38.317 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-22T21:46:38.393 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T21:46:38.469 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-22T21:46:38.548 INFO:teuthology.orchestra.run.trial034.stdout:Setting up zip (3.0-12build2) ... 2026-02-22T21:46:38.548 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T21:46:38.797 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2026-02-22T21:46:38.863 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2026-02-22T21:46:38.864 INFO:teuthology.orchestra.run.trial034.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-22T21:46:38.865 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T21:46:38.960 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T21:46:39.081 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2026-02-22T21:46:39.211 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T21:46:39.292 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-22T21:46:39.396 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-22T21:46:39.468 INFO:teuthology.orchestra.run.trial034.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-22T21:46:39.469 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:39.549 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T21:46:40.048 INFO:teuthology.orchestra.run.trial034.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2026-02-22T21:46:40.069 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T21:46:40.073 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-22T21:46:40.152 INFO:teuthology.orchestra.run.trial034.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T21:46:40.153 INFO:teuthology.orchestra.run.trial034.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-22T21:46:40.154 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-22T21:46:40.230 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-22T21:46:40.302 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T21:46:40.303 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-22T21:46:40.380 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2026-02-22T21:46:40.452 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-logutils (0.3.3-8) ... 2026-02-22T21:46:40.527 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-22T21:46:40.601 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2026-02-22T21:46:40.675 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-22T21:46:40.753 INFO:teuthology.orchestra.run.trial034.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T21:46:40.754 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-22T21:46:40.837 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T21:46:40.838 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T21:46:40.915 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-22T21:46:40.993 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T21:46:41.075 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-22T21:46:41.149 INFO:teuthology.orchestra.run.trial034.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-22T21:46:41.150 INFO:teuthology.orchestra.run.trial034.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2026-02-22T21:46:41.151 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T21:46:41.151 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T21:46:41.289 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2026-02-22T21:46:41.363 INFO:teuthology.orchestra.run.trial034.stdout:Setting up lua-any (27ubuntu1) ... 2026-02-22T21:46:41.364 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-22T21:46:41.439 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T21:46:41.440 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-22T21:46:41.514 INFO:teuthology.orchestra.run.trial034.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-22T21:46:41.514 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-webtest (2.0.35-1) ... 2026-02-22T21:46:41.600 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-22T21:46:41.732 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-pastescript (2.0.2-4) ... 2026-02-22T21:46:41.821 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2026-02-22T21:46:41.930 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T21:46:41.931 INFO:teuthology.orchestra.run.trial034.stdout:Setting up librados2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:41.932 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libsqlite3-mod-ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:41.933 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T21:46:42.467 INFO:teuthology.orchestra.run.trial034.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2026-02-22T21:46:42.479 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libcephfs2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:42.480 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libradosstriper1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:42.481 INFO:teuthology.orchestra.run.trial034.stdout:Setting up librbd1 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:42.482 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-mgr-modules-core (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:42.483 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:42.552 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T21:46:42.552 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T21:46:42.972 INFO:teuthology.orchestra.run.trial034.stdout:Setting up libcephfs-dev (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:42.973 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-rados (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:42.974 INFO:teuthology.orchestra.run.trial034.stdout:Setting up librgw2 (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:42.975 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-rbd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:42.976 INFO:teuthology.orchestra.run.trial034.stdout:Setting up rbd-fuse (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:42.977 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-rgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:42.978 INFO:teuthology.orchestra.run.trial034.stdout:Setting up python3-cephfs (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:42.978 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-common (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:43.007 INFO:teuthology.orchestra.run.trial034.stdout:Adding group ceph....done 2026-02-22T21:46:43.043 INFO:teuthology.orchestra.run.trial034.stdout:Adding system user ceph....done 2026-02-22T21:46:43.053 INFO:teuthology.orchestra.run.trial034.stdout:Setting system user ceph properties....done 2026-02-22T21:46:43.061 INFO:teuthology.orchestra.run.trial034.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-22T21:46:43.147 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-22T21:46:43.433 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-22T21:46:43.818 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-test (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:43.819 INFO:teuthology.orchestra.run.trial034.stdout:Setting up radosgw (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:44.085 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T21:46:44.085 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T21:46:44.461 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-base (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:44.578 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-22T21:46:44.978 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-mds (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:45.037 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T21:46:45.037 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T21:46:45.418 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-mgr (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:45.493 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T21:46:45.493 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T21:46:45.874 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-osd (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:45.948 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T21:46:45.948 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T21:46:46.329 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-mgr-k8sevents (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:46.330 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-mgr-diskprediction-local (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:46.340 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-mon (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:46.407 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T21:46:46.408 INFO:teuthology.orchestra.run.trial034.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T21:46:46.810 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-mgr-cephadm (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:46.821 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:46.822 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-mgr-dashboard (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:46.832 INFO:teuthology.orchestra.run.trial034.stdout:Setting up ceph-volume (19.2.3-689-gf90f4eae-1jammy) ... 2026-02-22T21:46:46.940 INFO:teuthology.orchestra.run.trial034.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2026-02-22T21:46:46.947 INFO:teuthology.orchestra.run.trial034.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-22T21:46:47.189 INFO:teuthology.orchestra.run.trial034.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-22T21:46:47.275 INFO:teuthology.orchestra.run.trial034.stdout:Processing triggers for install-info (6.8-4build1) ... 2026-02-22T21:46:47.761 INFO:teuthology.orchestra.run.trial034.stdout: 2026-02-22T21:46:47.761 INFO:teuthology.orchestra.run.trial034.stdout:Running kernel seems to be up-to-date. 2026-02-22T21:46:47.761 INFO:teuthology.orchestra.run.trial034.stdout: 2026-02-22T21:46:47.761 INFO:teuthology.orchestra.run.trial034.stdout:The processor microcode seems to be up-to-date. 2026-02-22T21:46:47.761 INFO:teuthology.orchestra.run.trial034.stdout: 2026-02-22T21:46:47.761 INFO:teuthology.orchestra.run.trial034.stdout:No services need to be restarted. 2026-02-22T21:46:47.761 INFO:teuthology.orchestra.run.trial034.stdout: 2026-02-22T21:46:47.761 INFO:teuthology.orchestra.run.trial034.stdout:No containers need to be restarted. 2026-02-22T21:46:47.761 INFO:teuthology.orchestra.run.trial034.stdout: 2026-02-22T21:46:47.761 INFO:teuthology.orchestra.run.trial034.stdout:No user sessions are running outdated binaries. 2026-02-22T21:46:47.761 INFO:teuthology.orchestra.run.trial034.stdout: 2026-02-22T21:46:47.761 INFO:teuthology.orchestra.run.trial034.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-22T21:46:48.564 INFO:teuthology.orchestra.run.trial034.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-22T21:46:48.570 DEBUG:teuthology.parallel:result is None 2026-02-22T21:46:48.571 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:46:48.753 DEBUG:teuthology.orchestra.run.trial034:> dpkg-query -W -f '${Version}' ceph 2026-02-22T21:46:48.764 INFO:teuthology.orchestra.run.trial034.stdout:19.2.3-689-gf90f4eae-1jammy 2026-02-22T21:46:48.764 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-689-gf90f4eae-1jammy 2026-02-22T21:46:48.764 INFO:teuthology.task.install:The correct ceph version 19.2.3-689-gf90f4eae-1jammy is installed. 2026-02-22T21:46:48.766 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-22T21:46:48.766 DEBUG:teuthology.orchestra.run.trial034:> set -ex 2026-02-22T21:46:48.766 DEBUG:teuthology.orchestra.run.trial034:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T21:46:48.815 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-22T21:46:48.815 DEBUG:teuthology.orchestra.run.trial034:> set -ex 2026-02-22T21:46:48.815 DEBUG:teuthology.orchestra.run.trial034:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T21:46:48.866 DEBUG:teuthology.orchestra.run.trial034:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T21:46:48.913 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-22T21:46:48.913 DEBUG:teuthology.orchestra.run.trial034:> set -ex 2026-02-22T21:46:48.913 DEBUG:teuthology.orchestra.run.trial034:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T21:46:48.962 DEBUG:teuthology.orchestra.run.trial034:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T21:46:49.014 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-22T21:46:49.015 DEBUG:teuthology.orchestra.run.trial034:> set -ex 2026-02-22T21:46:49.015 DEBUG:teuthology.orchestra.run.trial034:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T21:46:49.066 DEBUG:teuthology.orchestra.run.trial034:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T21:46:49.114 INFO:teuthology.run_tasks:Running task workunit... 2026-02-22T21:46:49.141 INFO:tasks.workunit:Pulling workunits from ref f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:46:49.142 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-22T21:46:49.142 INFO:tasks.workunit:timeout=3h 2026-02-22T21:46:49.142 INFO:tasks.workunit:cleanup=True 2026-02-22T21:46:49.142 DEBUG:teuthology.orchestra.run.trial034:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-22T21:46:49.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:46:49.157 INFO:teuthology.orchestra.run.trial034.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-22T21:46:49.157 DEBUG:teuthology.orchestra.run.trial034:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-22T21:46:49.202 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-22T21:46:49.203 DEBUG:teuthology.orchestra.run.trial034:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-22T21:46:49.250 DEBUG:teuthology.orchestra.run.trial034:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f90f4eaebb28b619cf76e80cbbe997656da69579 2026-02-22T21:46:49.301 INFO:tasks.workunit.client.0.trial034.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-22T21:47:11.806 INFO:tasks.workunit.client.0.trial034.stderr:Note: switching to 'f90f4eaebb28b619cf76e80cbbe997656da69579'. 2026-02-22T21:47:11.807 INFO:tasks.workunit.client.0.trial034.stderr: 2026-02-22T21:47:11.807 INFO:tasks.workunit.client.0.trial034.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-22T21:47:11.807 INFO:tasks.workunit.client.0.trial034.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-22T21:47:11.807 INFO:tasks.workunit.client.0.trial034.stderr:state without impacting any branches by switching back to a branch. 2026-02-22T21:47:11.807 INFO:tasks.workunit.client.0.trial034.stderr: 2026-02-22T21:47:11.807 INFO:tasks.workunit.client.0.trial034.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-22T21:47:11.807 INFO:tasks.workunit.client.0.trial034.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-22T21:47:11.807 INFO:tasks.workunit.client.0.trial034.stderr: 2026-02-22T21:47:11.808 INFO:tasks.workunit.client.0.trial034.stderr: git switch -c 2026-02-22T21:47:11.808 INFO:tasks.workunit.client.0.trial034.stderr: 2026-02-22T21:47:11.808 INFO:tasks.workunit.client.0.trial034.stderr:Or undo this operation with: 2026-02-22T21:47:11.808 INFO:tasks.workunit.client.0.trial034.stderr: 2026-02-22T21:47:11.808 INFO:tasks.workunit.client.0.trial034.stderr: git switch - 2026-02-22T21:47:11.808 INFO:tasks.workunit.client.0.trial034.stderr: 2026-02-22T21:47:11.808 INFO:tasks.workunit.client.0.trial034.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-22T21:47:11.808 INFO:tasks.workunit.client.0.trial034.stderr: 2026-02-22T21:47:11.808 INFO:tasks.workunit.client.0.trial034.stderr:HEAD is now at f90f4eaebb2 Merge branch 'wip-74712-squid' of https://github.com/idryomov/ceph into wip-yuri6-testing-2026-02-22-1629-squid 2026-02-22T21:47:11.817 DEBUG:teuthology.orchestra.run.trial034:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-22T21:47:11.868 DEBUG:teuthology.orchestra.run.trial034:> set -ex 2026-02-22T21:47:11.868 DEBUG:teuthology.orchestra.run.trial034:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-22T21:47:11.915 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2026-02-22T21:47:11.915 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2026-02-22T21:47:11.916 DEBUG:teuthology.orchestra.run.trial034:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2026-02-22T21:47:11.966 INFO:tasks.workunit.client.0.trial034.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2026-02-22T21:47:11.966 INFO:tasks.workunit.client.0.trial034.stderr:++ TIMEOUT=300 2026-02-22T21:47:11.966 INFO:tasks.workunit.client.0.trial034.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2026-02-22T21:47:11.966 INFO:tasks.workunit.client.0.trial034.stderr:++ MAX_TIMEOUT=15 2026-02-22T21:47:11.966 INFO:tasks.workunit.client.0.trial034.stderr:++ PG_NUM=4 2026-02-22T21:47:11.967 INFO:tasks.workunit.client.0.trial034.stderr:++ TMPDIR=/tmp 2026-02-22T21:47:11.967 INFO:tasks.workunit.client.0.trial034.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2026-02-22T21:47:11.967 INFO:tasks.workunit.client.0.trial034.stderr:++ TESTDIR=/home/ubuntu/cephtest 2026-02-22T21:47:11.967 INFO:tasks.workunit.client.0.trial034.stderr:++ type xmlstarlet 2026-02-22T21:47:11.967 INFO:tasks.workunit.client.0.trial034.stderr:++ XMLSTARLET=xmlstarlet 2026-02-22T21:47:11.967 INFO:tasks.workunit.client.0.trial034.stderr:+++ uname 2026-02-22T21:47:11.968 INFO:tasks.workunit.client.0.trial034.stderr:++ '[' Linux = FreeBSD ']' 2026-02-22T21:47:11.968 INFO:tasks.workunit.client.0.trial034.stderr:++ SED=sed 2026-02-22T21:47:11.968 INFO:tasks.workunit.client.0.trial034.stderr:++ AWK=awk 2026-02-22T21:47:11.968 INFO:tasks.workunit.client.0.trial034.stderr:+++ stty -a 2026-02-22T21:47:11.968 INFO:tasks.workunit.client.0.trial034.stderr:+++ head -1 2026-02-22T21:47:11.968 INFO:tasks.workunit.client.0.trial034.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2026-02-22T21:47:11.969 INFO:tasks.workunit.client.0.trial034.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T21:47:11.969 INFO:tasks.workunit.client.0.trial034.stderr:++ termwidth= 2026-02-22T21:47:11.969 INFO:tasks.workunit.client.0.trial034.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2026-02-22T21:47:11.969 INFO:tasks.workunit.client.0.trial034.stderr:++ DIFFCOLOPTS='-y ' 2026-02-22T21:47:11.970 INFO:tasks.workunit.client.0.trial034.stderr:++ KERNCORE=kernel.core_pattern 2026-02-22T21:47:11.970 INFO:tasks.workunit.client.0.trial034.stderr:++ EXTRA_OPTS= 2026-02-22T21:47:11.970 INFO:tasks.workunit.client.0.trial034.stderr:++ test '' = TESTS 2026-02-22T21:47:11.971 INFO:tasks.workunit.client.0.trial034.stderr:++ uname -m 2026-02-22T21:47:11.975 INFO:tasks.workunit.client.0.trial034.stderr:+ arch=x86_64 2026-02-22T21:47:11.975 INFO:tasks.workunit.client.0.trial034.stderr:+ case $arch in 2026-02-22T21:47:11.976 INFO:tasks.workunit.client.0.trial034.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2026-02-22T21:47:11.976 INFO:tasks.workunit.client.0.trial034.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2026-02-22T21:47:11.976 INFO:tasks.workunit.client.0.trial034.stderr:+ plugins=(jerasure shec lrc isa) 2026-02-22T21:47:11.976 INFO:tasks.workunit.client.0.trial034.stderr:+ main test-erasure-code-plugins 2026-02-22T21:47:11.976 INFO:tasks.workunit.client.0.trial034.stderr:+ local dir=td/test-erasure-code-plugins 2026-02-22T21:47:11.977 INFO:tasks.workunit.client.0.trial034.stderr:+ shift 2026-02-22T21:47:11.977 INFO:tasks.workunit.client.0.trial034.stderr:+ shopt -s -o xtrace 2026-02-22T21:47:11.977 INFO:tasks.workunit.client.0.trial034.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T21:47:11.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T21:47:11.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T21:47:11.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: export PYTHONWARNINGS=ignore 2026-02-22T21:47:11.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: PYTHONWARNINGS=ignore 2026-02-22T21:47:11.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: export CEPH_CONF=/dev/null 2026-02-22T21:47:11.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: CEPH_CONF=/dev/null 2026-02-22T21:47:11.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: main: unset CEPH_ARGS 2026-02-22T21:47:11.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: main: local code 2026-02-22T21:47:11.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: main: run td/test-erasure-code-plugins 2026-02-22T21:47:11.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:11.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2026-02-22T21:47:11.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2026-02-22T21:47:11.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2026-02-22T21:47:11.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2026-02-22T21:47:11.978 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2026-02-22T21:47:11.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none ' 2026-02-22T21:47:11.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2026-02-22T21:47:11.979 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2026-02-22T21:47:11.979 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T21:47:11.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_warning 2026-02-22T21:47:11.981 INFO:tasks.workunit.client.0.trial034.stderr:TEST_preload_no_warning 2026-02-22T21:47:11.981 INFO:tasks.workunit.client.0.trial034.stderr:TEST_preload_no_warning_default 2026-02-22T21:47:11.982 INFO:tasks.workunit.client.0.trial034.stderr:TEST_preload_warning' 2026-02-22T21:47:11.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-22T21:47:11.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2026-02-22T21:47:11.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:11.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2026-02-22T21:47:11.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:11.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2026-02-22T21:47:11.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:11.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:47:11.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:47:11.983 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:47:11.983 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:47:11.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:47:11.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:47:11.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:47:11.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:47:11.984 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:47:11.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:47:11.984 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:47:11.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:47:11.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:47:11.987 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:47:11.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:47:11.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:47:11.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:47:11.987 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:47:11.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:47:11.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:47:11.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:47:11.988 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:47:11.988 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:11.988 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:11.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:47:11.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:47:11.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:47:11.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2026-02-22T21:47:11.991 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:47:11.991 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:11.991 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:11.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12473 2026-02-22T21:47:11.992 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:47:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2026-02-22T21:47:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2026-02-22T21:47:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:47:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-22T21:47:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2026-02-22T21:47:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:47:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:47:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:47:11.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-22T21:47:11.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2026-02-22T21:47:12.027 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:47:12.027 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:47:12.027 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:47:12.027 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:47:12.027 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:12.027 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:12.027 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:12.028 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T21:47:12.057 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:47:12.058 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:47:12.058 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:47:12.058 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:47:12.058 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:47:12.059 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:47:12.060 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:47:12.060 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:12.060 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:12.060 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:12.060 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:12.060 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:12.060 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:12.060 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:47:12.060 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get fsid 2026-02-22T21:47:12.125 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:47:12.125 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:47:12.125 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:47:12.126 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:47:12.126 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:47:12.126 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:47:12.126 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:12.126 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:12.127 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:12.127 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:12.127 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:12.127 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:12.127 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:47:12.127 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get mon_host 2026-02-22T21:47:12.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2026-02-22T21:47:12.183 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:12.183 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:47:12.183 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:47:12.183 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:47:12.183 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-22T21:47:12.183 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:47:12.290 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:47:12.291 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:47:12.291 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:47:12.291 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:47:12.291 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:12.291 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:12.291 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:12.291 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:47:12.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:47:12.311 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2026-02-22T21:47:12.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:47:12.313 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:47:12.314 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:47:12.314 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:47:12.314 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:47:12.314 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:12.314 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:12.314 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:12.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:12.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:47:12.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:47:12.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:47:12.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:47:12.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:47:12.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:47:12.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:47:12.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:47:12.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:47:12.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:47:12.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:47:12.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-22T21:47:12.316 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:47:12.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3c662420-60ed-492e-94ff-f48f885683f1 2026-02-22T21:47:12.316 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 3c662420-60ed-492e-94ff-f48f885683f1 2026-02-22T21:47:12.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3c662420-60ed-492e-94ff-f48f885683f1' 2026-02-22T21:47:12.316 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:47:12.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBgeZtplrm0EhAA9KwoAEo1RrMhT+JjYovweg== 2026-02-22T21:47:12.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBgeZtplrm0EhAA9KwoAEo1RrMhT+JjYovweg=="}' 2026-02-22T21:47:12.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3c662420-60ed-492e-94ff-f48f885683f1 -i td/test-erasure-code-plugins/0/new.json 2026-02-22T21:47:12.430 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:47:12.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-22T21:47:12.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBgeZtplrm0EhAA9KwoAEo1RrMhT+JjYovweg== --osd-uuid 3c662420-60ed-492e-94ff-f48f885683f1 2026-02-22T21:47:12.453 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:12.437+0000 7f44979df8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:12.454 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:12.441+0000 7f44979df8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:12.455 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:12.441+0000 7f44979df8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:12.456 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:12.441+0000 7f44979df8c0 -1 bdev(0x5636e757fc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:47:12.456 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:12.441+0000 7f44979df8c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-22T21:47:15.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-22T21:47:15.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:47:15.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:47:15.237 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:47:15.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:47:15.510 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:47:15.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:47:15.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:47:15.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:47:15.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:47:15.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:47:15.533 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:15.517+0000 7f62967548c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:15.534 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:15.521+0000 7f62967548c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:15.535 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:15.521+0000 7f62967548c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:15.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:47:15.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:47:15.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:47:15.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:47:15.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:47:15.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:15.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:47:15.704 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:47:15.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:15.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:15.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:16.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:16.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:16.908 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:47:16.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:47:16.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:16.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:17.060 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:17.045+0000 7f62967548c0 -1 Falling back to public interface 2026-02-22T21:47:17.108 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:18.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:18.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:18.109 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:47:18.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:47:18.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:18.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:18.309 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:19.310 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:19.310 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:19.310 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:47:19.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:47:19.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:19.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:19.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:20.084 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:20.069+0000 7f62967548c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:47:20.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:20.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:20.514 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:47:20.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:47:20.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:20.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:20.710 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1923337433,v1:127.0.0.1:6803/1923337433] [v2:127.0.0.1:6804/1923337433,v1:127.0.0.1:6805/1923337433] exists,up 3c662420-60ed-492e-94ff-f48f885683f1 2026-02-22T21:47:20.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:47:20.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:47:20.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:47:20.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2026-02-22T21:47:20.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2026-02-22T21:47:20.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:20.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:47:20.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T21:47:20.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:47:20.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/1 2026-02-22T21:47:20.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:47:20.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:47:20.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:47:20.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:47:20.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2026-02-22T21:47:20.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2026-02-22T21:47:20.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:47:20.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:47:20.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:47:20.715 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:47:20.715 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:47:20.715 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:47:20.715 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:47:20.715 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:47:20.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:47:20.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:47:20.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/1 2026-02-22T21:47:20.717 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:47:20.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a402b276-3e12-4f10-9bbc-24d87a5f7c90 2026-02-22T21:47:20.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a402b276-3e12-4f10-9bbc-24d87a5f7c90' 2026-02-22T21:47:20.717 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 a402b276-3e12-4f10-9bbc-24d87a5f7c90 2026-02-22T21:47:20.718 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:47:20.725 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBoeZtpvEuSKhAA30SmGg8pzzAt6Bk8U5TnEw== 2026-02-22T21:47:20.725 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBoeZtpvEuSKhAA30SmGg8pzzAt6Bk8U5TnEw=="}' 2026-02-22T21:47:20.726 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a402b276-3e12-4f10-9bbc-24d87a5f7c90 -i td/test-erasure-code-plugins/1/new.json 2026-02-22T21:47:20.927 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:47:20.936 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/1/new.json 2026-02-22T21:47:20.937 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBoeZtpvEuSKhAA30SmGg8pzzAt6Bk8U5TnEw== --osd-uuid a402b276-3e12-4f10-9bbc-24d87a5f7c90 2026-02-22T21:47:20.953 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:20.937+0000 7fc651c7a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:20.954 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:20.941+0000 7fc651c7a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:20.955 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:20.941+0000 7fc651c7a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:20.956 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:20.941+0000 7fc651c7a8c0 -1 bdev(0x561f79fc7c00 td/test-erasure-code-plugins/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:47:20.956 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:20.941+0000 7fc651c7a8c0 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2026-02-22T21:47:23.243 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2026-02-22T21:47:23.243 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:47:23.244 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T21:47:23.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:47:23.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:47:23.521 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T21:47:23.522 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T21:47:23.523 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:47:23.523 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:47:23.523 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:47:23.523 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:47:23.543 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:23.529+0000 7f74e154b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:23.545 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:23.529+0000 7f74e154b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:23.546 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:23.529+0000 7f74e154b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:23.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T21:47:23.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:47:23.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T21:47:23.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:47:23.724 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:47:23.724 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:23.724 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:47:23.724 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:47:23.724 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:23.725 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:47:23.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:24.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:24.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:24.923 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:47:24.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:47:24.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:24.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:47:25.027 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:25.013+0000 7f74e154b8c0 -1 Falling back to public interface 2026-02-22T21:47:25.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:26.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:26.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:26.116 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:47:26.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:47:26.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:26.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:47:26.317 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:27.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:27.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:27.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:47:27.318 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:47:27.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:27.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:47:27.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:28.291 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:28.277+0000 7f74e154b8c0 -1 osd.1 0 log_to_monitors true 2026-02-22T21:47:28.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:28.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:28.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:47:28.519 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:47:28.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:28.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:47:28.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:29.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:29.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:29.712 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:47:29.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:47:29.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:29.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:47:29.910 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3678292828,v1:127.0.0.1:6811/3678292828] [v2:127.0.0.1:6812/3678292828,v1:127.0.0.1:6813/3678292828] exists,up a402b276-3e12-4f10-9bbc-24d87a5f7c90 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/2 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:47:29.911 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:47:29.912 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:47:29.912 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:47:29.912 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:47:29.912 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:29.912 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:29.912 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:47:29.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/2 2026-02-22T21:47:29.914 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:47:29.916 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6cdf2a06-6d68-4645-a301-b43fbd35624d 2026-02-22T21:47:29.916 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6cdf2a06-6d68-4645-a301-b43fbd35624d' 2026-02-22T21:47:29.916 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 6cdf2a06-6d68-4645-a301-b43fbd35624d 2026-02-22T21:47:29.916 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:47:29.928 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxeZtp1aaqNhAA2lXhrsysMvf9ZBbjsov6hQ== 2026-02-22T21:47:29.928 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxeZtp1aaqNhAA2lXhrsysMvf9ZBbjsov6hQ=="}' 2026-02-22T21:47:29.928 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6cdf2a06-6d68-4645-a301-b43fbd35624d -i td/test-erasure-code-plugins/2/new.json 2026-02-22T21:47:30.135 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:47:30.144 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/2/new.json 2026-02-22T21:47:30.144 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxeZtp1aaqNhAA2lXhrsysMvf9ZBbjsov6hQ== --osd-uuid 6cdf2a06-6d68-4645-a301-b43fbd35624d 2026-02-22T21:47:30.159 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:30.145+0000 7f7499b568c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:30.161 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:30.145+0000 7f7499b568c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:30.162 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:30.145+0000 7f7499b568c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:30.162 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:30.149+0000 7f7499b568c0 -1 bdev(0x558318d59c00 td/test-erasure-code-plugins/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:47:30.162 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:30.149+0000 7f7499b568c0 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2026-02-22T21:47:32.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2026-02-22T21:47:32.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:47:32.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:47:32.467 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T21:47:32.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:47:32.738 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T21:47:32.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T21:47:32.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:47:32.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:47:32.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:47:32.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:47:32.759 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:32.745+0000 7f73862148c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:32.761 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:32.745+0000 7f73862148c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:32.762 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:32.745+0000 7f73862148c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:32.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T21:47:32.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:47:32.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T21:47:32.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:47:32.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:47:32.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:32.932 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:47:32.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:47:32.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:32.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:47:33.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:34.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:34.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:34.136 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:47:34.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:47:34.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:34.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:47:34.276 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:34.261+0000 7f73862148c0 -1 Falling back to public interface 2026-02-22T21:47:34.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:35.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:35.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:35.349 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:47:35.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:47:35.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:35.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:47:35.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:36.547 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:36.547 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:36.547 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:47:36.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:47:36.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:36.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:47:36.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:37.313 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:37.297+0000 7f73862148c0 -1 osd.2 0 log_to_monitors true 2026-02-22T21:47:37.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:37.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:37.746 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:47:37.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:47:37.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:37.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:47:37.939 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4063237592,v1:127.0.0.1:6819/4063237592] [v2:127.0.0.1:6820/4063237592,v1:127.0.0.1:6821/4063237592] exists,up 6cdf2a06-6d68-4645-a301-b43fbd35624d 2026-02-22T21:47:37.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:47:37.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:47:37.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:47:37.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2026-02-22T21:47:37.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T21:47:38.132 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' does not exist 2026-02-22T21:47:38.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T21:47:38.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T21:47:38.381 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T21:47:38.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:47:39.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T21:47:39.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2026-02-22T21:47:39.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:47:39.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:47:39.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:47:39.679 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:47:39.679 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:47:39.679 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:47:39.680 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:47:39.680 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:47:39.680 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:47:39.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:47:39.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:47:39.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:47:39.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:47:39.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:47:39.785 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:47:39.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:47:39.980 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:47:39.980 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T21:47:39.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:47:39.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:47:39.980 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:47:40.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836486 2026-02-22T21:47:40.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836486 2026-02-22T21:47:40.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836486' 2026-02-22T21:47:40.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:47:40.056 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:47:40.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=38654705669 2026-02-22T21:47:40.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 38654705669 2026-02-22T21:47:40.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836486 1-38654705669' 2026-02-22T21:47:40.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:47:40.131 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:47:40.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=55834574851 2026-02-22T21:47:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 55834574851 2026-02-22T21:47:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836486 1-38654705669 2-55834574851' 2026-02-22T21:47:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:47:40.208 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836486 2026-02-22T21:47:40.208 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:47:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:47:40.209 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836486 2026-02-22T21:47:40.209 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:47:40.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836486 2026-02-22T21:47:40.211 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 21474836486 2026-02-22T21:47:40.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-22T21:47:40.211 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:47:40.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836484 -lt 21474836486 2026-02-22T21:47:40.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:47:41.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:47:41.404 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:47:41.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-22T21:47:41.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:47:41.602 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-38654705669 2026-02-22T21:47:41.603 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:47:41.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:47:41.604 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-38654705669 2026-02-22T21:47:41.604 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:47:41.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=38654705669 2026-02-22T21:47:41.606 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 38654705669 2026-02-22T21:47:41.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-22T21:47:41.607 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:47:41.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-22T21:47:41.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:47:41.805 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-55834574851 2026-02-22T21:47:41.805 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:47:41.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:47:41.806 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-55834574851 2026-02-22T21:47:41.807 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:47:41.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=55834574851 2026-02-22T21:47:41.808 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 55834574851 2026-02-22T21:47:41.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 55834574851' 2026-02-22T21:47:41.809 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:47:42.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 55834574851 -lt 55834574851 2026-02-22T21:47:42.010 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:47:42.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:47:42.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:47:42.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T21:47:42.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:47:42.263 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:47:42.264 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:47:42.264 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:47:42.264 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:47:42.264 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:47:42.264 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:47:42.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T21:47:42.453 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:47:42.453 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:47:42.453 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:47:42.703 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T21:47:42.703 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:47:42.703 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:47:42.703 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2026-02-22T21:47:42.703 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2026-02-22T21:47:42.931 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-22T21:47:42.932 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:42.932 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:42.932 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:42.932 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:42.932 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:42.932 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:42.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-22T21:47:42.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:47:42.981 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:47:42.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:47:42.989 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:42.833+0000 7f97a6ae3640 0 mon.a@0(leader).osd e20 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2026-02-22T21:47:42.989 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:42.905+0000 7f97a6ae3640 0 mon.a@0(leader).osd e21 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2026-02-22T21:47:42.989 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:42.905+0000 7f97a6ae3640 0 mon.a@0(leader).osd e21 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2026-02-22T21:47:42.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2026-02-22T21:47:42.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2026-02-22T21:47:43.248 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-22T21:47:43.248 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:43.248 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:43.248 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:43.248 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:43.248 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:43.248 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:43.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-22T21:47:43.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:47:43.297 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:47:43.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:47:43.305 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:43.121+0000 7f97a6ae3640 0 mon.a@0(leader).osd e21 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2026-02-22T21:47:43.306 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:43.221+0000 7f97a6ae3640 0 mon.a@0(leader).osd e22 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2026-02-22T21:47:43.306 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:43.221+0000 7f97a6ae3640 0 mon.a@0(leader).osd e22 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2026-02-22T21:47:43.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2026-02-22T21:47:43.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2026-02-22T21:47:43.577 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-22T21:47:43.577 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:43.577 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:43.577 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:43.578 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:43.578 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:43.578 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:43.578 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-22T21:47:43.578 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:47:43.634 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:47:43.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:47:43.654 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:43.437+0000 7f97a6ae3640 0 mon.a@0(leader).osd e22 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2026-02-22T21:47:43.654 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:43.545+0000 7f97a6ae3640 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2026-02-22T21:47:43.654 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:43.545+0000 7f97a6ae3640 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2026-02-22T21:47:43.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2026-02-22T21:47:43.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2026-02-22T21:47:43.920 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-22T21:47:43.920 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:43.920 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:43.920 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:43.920 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:43.921 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:43.921 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:43.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-22T21:47:43.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:47:43.968 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:47:43.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:47:43.976 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:43.789+0000 7f97a6ae3640 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2026-02-22T21:47:43.977 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:43.893+0000 7f97a6ae3640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2026-02-22T21:47:43.977 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:43.893+0000 7f97a6ae3640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2026-02-22T21:47:43.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2026-02-22T21:47:43.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2026-02-22T21:47:44.236 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-22T21:47:44.236 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:44.236 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:44.236 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:44.236 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:44.236 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:44.236 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:44.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-22T21:47:44.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:47:44.292 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:47:44.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:47:44.300 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:44.105+0000 7f97a6ae3640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2026-02-22T21:47:44.301 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:44.209+0000 7f97a6ae3640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2026-02-22T21:47:44.301 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:44.209+0000 7f97a6ae3640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2026-02-22T21:47:44.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2026-02-22T21:47:44.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2026-02-22T21:47:44.560 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2026-02-22T21:47:44.560 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:44.560 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:44.561 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:44.561 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:44.561 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:44.561 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:44.561 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2026-02-22T21:47:44.561 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:47:44.608 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:47:44.614 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:47:44.617 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:44.433+0000 7f97a6ae3640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2026-02-22T21:47:44.617 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:44.533+0000 7f97a6ae3640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2026-02-22T21:47:44.617 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:47:44.533+0000 7f97a6ae3640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2026-02-22T21:47:44.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2026-02-22T21:47:44.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:44.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:47:44.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:47:44.619 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:47:44.619 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:47:44.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:47:44.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:47:44.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:47:44.726 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:47:44.727 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:47:44.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:47:44.727 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:47:44.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:47:44.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:47:44.729 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:47:44.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:47:44.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:47:44.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:47:44.730 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:47:44.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:47:44.732 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:47:44.732 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:47:44.743 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:47:44.743 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:44.743 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:44.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:47:44.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:47:44.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:47:44.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-22T21:47:44.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2026-02-22T21:47:44.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:44.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2026-02-22T21:47:44.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2026-02-22T21:47:44.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:44.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2026-02-22T21:47:44.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:44.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:47:44.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:47:44.746 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:47:44.746 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:47:44.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:47:44.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:47:44.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:47:44.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:47:44.747 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:47:44.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:47:44.748 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:47:44.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:47:44.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:47:44.750 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:47:44.751 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:47:44.751 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:47:44.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:47:44.752 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:47:44.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:47:44.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:47:44.754 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:47:44.755 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:47:44.755 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:44.755 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:44.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:47:44.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:47:44.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:47:44.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2026-02-22T21:47:44.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:47:44.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:44.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:44.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12473 2026-02-22T21:47:44.760 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:47:44.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:47:44.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:47:44.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-22T21:47:44.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2026-02-22T21:47:44.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:44.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:47:44.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:47:44.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:47:44.762 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-22T21:47:44.762 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2026-02-22T21:47:44.781 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:47:44.781 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:47:44.781 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:47:44.782 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:47:44.782 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:44.782 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:44.782 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:44.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2026-02-22T21:47:44.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:47:44.809 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:47:44.809 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:47:44.809 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:47:44.810 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:47:44.810 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:47:44.811 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:47:44.811 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:44.811 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:44.811 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:44.811 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:44.811 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:44.812 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:44.812 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:47:44.812 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get fsid 2026-02-22T21:47:44.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:47:44.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:47:44.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:47:44.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:47:44.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:47:44.869 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:47:44.870 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:44.870 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:44.870 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:44.870 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:44.870 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:44.870 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:44.871 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:47:44.871 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get mon_host 2026-02-22T21:47:44.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2026-02-22T21:47:44.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:44.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:47:44.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:47:44.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:47:44.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-22T21:47:44.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:47:45.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:47:45.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:47:45.035 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:47:45.035 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:47:45.035 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:45.035 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:45.035 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:45.035 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:47:45.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:47:45.054 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2026-02-22T21:47:45.054 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:45.054 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:45.055 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:45.055 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:45.055 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:45.055 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:45.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2026-02-22T21:47:45.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:47:45.104 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:47:45.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2026-02-22T21:47:45.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:45.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:47:45.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:47:45.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:47:45.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-22T21:47:45.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:47:45.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:47:45.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:47:45.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:47:45.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-22T21:47:45.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-22T21:47:45.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:47:45.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:47:45.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:47:45.116 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:47:45.116 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:47:45.116 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:47:45.116 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:47:45.116 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:45.116 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:45.116 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:45.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:45.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:47:45.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=36a36c18-f730-4dc7-a147-c4c72e327715 2026-02-22T21:47:45.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 36a36c18-f730-4dc7-a147-c4c72e327715' 2026-02-22T21:47:45.118 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 36a36c18-f730-4dc7-a147-c4c72e327715 2026-02-22T21:47:45.118 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:47:45.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBeZtptmnOBhAATfvaQqFNgFrGPb6kRKa1fQ== 2026-02-22T21:47:45.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBeZtptmnOBhAATfvaQqFNgFrGPb6kRKa1fQ=="}' 2026-02-22T21:47:45.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 36a36c18-f730-4dc7-a147-c4c72e327715 -i td/test-erasure-code-plugins/0/new.json 2026-02-22T21:47:45.231 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:47:45.239 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-22T21:47:45.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQCBeZtptmnOBhAATfvaQqFNgFrGPb6kRKa1fQ== --osd-uuid 36a36c18-f730-4dc7-a147-c4c72e327715 2026-02-22T21:47:45.255 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:45.241+0000 7f39bf2828c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:45.256 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:45.241+0000 7f39bf2828c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:45.257 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:45.241+0000 7f39bf2828c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:45.257 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:45.241+0000 7f39bf2828c0 -1 bdev(0x559ef7905c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:47:45.258 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:45.241+0000 7f39bf2828c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-22T21:47:47.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-22T21:47:47.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:47:47.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:47:47.813 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:47:47.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:47:48.086 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:47:48.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:47:48.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2026-02-22T21:47:48.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:47:48.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:47:48.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:47:48.109 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:48.093+0000 7fbb52a868c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:48.111 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:48.097+0000 7fbb52a868c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:48.112 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:48.097+0000 7fbb52a868c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:48.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:47:48.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:47:48.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:47:48.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:47:48.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:47:48.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:48.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:47:48.283 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:47:48.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:48.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:48.478 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:49.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:49.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:49.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:47:49.481 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:47:49.482 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:49.482 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:49.583 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:49.569+0000 7fbb52a868c0 -1 Falling back to public interface 2026-02-22T21:47:49.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:50.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:50.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:50.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:47:50.682 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:47:50.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:50.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:50.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:51.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:51.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:51.884 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:47:51.885 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:47:51.885 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:51.885 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:52.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:52.793 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:52.777+0000 7fbb52a868c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:47:53.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:53.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:53.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:47:53.082 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:47:53.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:53.083 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:53.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:54.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:54.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:54.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:47:54.292 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:47:54.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:54.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:54.490 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4258664479,v1:127.0.0.1:6803/4258664479] [v2:127.0.0.1:6804/4258664479,v1:127.0.0.1:6805/4258664479] exists,up 36a36c18-f730-4dc7-a147-c4c72e327715 2026-02-22T21:47:54.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:47:54.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:47:54.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:47:54.493 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2026-02-22T21:47:54.493 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:47:54.493 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:47:54.493 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:54.493 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:54.493 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:54.493 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-osd.0.asok 2026-02-22T21:47:54.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2026-02-22T21:47:54.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-osd.0.asok log flush 2026-02-22T21:47:54.545 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:47:54.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:47:54.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2026-02-22T21:47:54.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2026-02-22T21:47:54.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:54.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:47:54.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:47:54.555 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:47:54.555 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:47:54.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:47:54.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:47:54.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:47:54.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:47:54.662 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:47:54.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:47:54.663 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:47:54.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:47:54.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:47:54.665 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:47:54.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:47:54.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:47:54.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:47:54.666 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:47:54.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:47:54.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:47:54.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:47:54.673 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:47:54.673 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:54.673 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:54.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:47:54.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:47:54.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:47:54.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2026-02-22T21:47:54.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2026-02-22T21:47:54.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:54.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2026-02-22T21:47:54.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:54.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:47:54.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:47:54.676 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:47:54.676 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:47:54.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:47:54.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:47:54.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:47:54.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:47:54.678 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:47:54.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:47:54.679 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:47:54.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:47:54.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:47:54.680 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:47:54.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:47:54.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:47:54.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:47:54.682 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:47:54.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:47:54.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:47:54.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:47:54.685 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:47:54.686 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:54.686 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:54.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:47:54.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:47:54.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:47:54.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2026-02-22T21:47:54.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:47:54.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:54.689 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:54.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12473 2026-02-22T21:47:54.690 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:47:54.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:47:54.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:47:54.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-22T21:47:54.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2026-02-22T21:47:54.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:54.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:47:54.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:47:54.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:47:54.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-22T21:47:54.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2026-02-22T21:47:54.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:47:54.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:47:54.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:47:54.712 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:47:54.712 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:54.712 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:54.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:54.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2026-02-22T21:47:54.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:47:54.738 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:47:54.738 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:47:54.739 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:47:54.739 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:47:54.740 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:47:54.740 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:47:54.741 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:54.741 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:54.741 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:54.741 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:54.741 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:54.741 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:54.741 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:47:54.741 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get fsid 2026-02-22T21:47:54.795 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:47:54.795 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:47:54.795 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:47:54.795 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:47:54.795 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:47:54.795 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:47:54.795 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:54.795 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:54.796 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:54.797 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:54.797 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:54.797 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:54.797 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:47:54.797 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get mon_host 2026-02-22T21:47:54.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2026-02-22T21:47:54.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:54.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:47:54.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:47:54.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:47:54.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-22T21:47:54.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:47:54.955 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:47:54.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:47:54.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:47:54.956 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:47:54.956 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:54.956 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:54.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:54.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:47:54.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:47:54.975 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2026-02-22T21:47:54.975 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:47:54.975 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:47:54.975 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:47:54.975 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:54.975 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:54.975 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:47:54.975 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2026-02-22T21:47:54.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:47:55.024 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:47:55.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2026-02-22T21:47:55.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:47:55.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:47:55.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:47:55.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:47:55.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-22T21:47:55.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:47:55.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:47:55.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:47:55.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:47:55.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-22T21:47:55.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-22T21:47:55.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:47:55.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:47:55.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:47:55.032 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:47:55.032 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:47:55.032 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2026-02-22T21:47:55.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-22T21:47:55.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:47:55.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a0e289e0-237b-4f44-8ab8-45e4076be7dc 2026-02-22T21:47:55.035 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 a0e289e0-237b-4f44-8ab8-45e4076be7dc 2026-02-22T21:47:55.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a0e289e0-237b-4f44-8ab8-45e4076be7dc' 2026-02-22T21:47:55.035 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:47:55.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLeZtpZz4bAhAAp6K2DMCYl8GEXGF0cKRVcg== 2026-02-22T21:47:55.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLeZtpZz4bAhAAp6K2DMCYl8GEXGF0cKRVcg=="}' 2026-02-22T21:47:55.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a0e289e0-237b-4f44-8ab8-45e4076be7dc -i td/test-erasure-code-plugins/0/new.json 2026-02-22T21:47:55.157 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:47:55.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-22T21:47:55.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQCLeZtpZz4bAhAAp6K2DMCYl8GEXGF0cKRVcg== --osd-uuid a0e289e0-237b-4f44-8ab8-45e4076be7dc 2026-02-22T21:47:55.180 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:55.164+0000 7ff275fb38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:55.181 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:55.164+0000 7ff275fb38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:55.183 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:55.168+0000 7ff275fb38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:55.183 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:55.168+0000 7ff275fb38c0 -1 bdev(0x55dc9b17fc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:47:55.183 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:55.168+0000 7ff275fb38c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-22T21:47:57.763 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-22T21:47:57.763 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:47:57.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:47:57.764 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:47:57.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:47:58.028 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:47:58.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:47:58.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2026-02-22T21:47:58.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:47:58.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:47:58.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:47:58.050 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:58.036+0000 7ff6f290d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:58.052 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:58.036+0000 7ff6f290d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:58.053 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:58.036+0000 7ff6f290d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:47:58.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:47:58.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:47:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:47:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:47:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:47:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:58.226 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:47:58.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:47:58.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:58.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:58.416 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:47:59.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:47:59.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:47:59.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:47:59.418 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:47:59.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:47:59.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:47:59.588 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:47:59.572+0000 7ff6f290d8c0 -1 Falling back to public interface 2026-02-22T21:47:59.614 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:00.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:00.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:00.616 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:48:00.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:48:00.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:00.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:00.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:01.815 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:01.815 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:01.816 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:48:01.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:48:01.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:01.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:02.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:02.610 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:02.596+0000 7ff6f290d8c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:48:03.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:03.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:03.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:48:03.009 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:48:03.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:03.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:03.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:04.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:04.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:04.202 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:48:04.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:48:04.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:04.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:04.393 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2910440205,v1:127.0.0.1:6803/2910440205] [v2:127.0.0.1:6804/2910440205,v1:127.0.0.1:6805/2910440205] exists,up a0e289e0-237b-4f44-8ab8-45e4076be7dc 2026-02-22T21:48:04.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:48:04.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:48:04.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:48:04.395 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2026-02-22T21:48:04.395 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:48:04.395 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:48:04.395 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:04.395 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:04.395 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:04.395 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-osd.0.asok 2026-02-22T21:48:04.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2026-02-22T21:48:04.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-osd.0.asok log flush 2026-02-22T21:48:04.452 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:48:04.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:48:04.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2026-02-22T21:48:04.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2026-02-22T21:48:04.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:04.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:48:04.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:48:04.461 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:48:04.461 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:48:04.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:48:04.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:48:04.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:48:04.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:48:04.571 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:48:04.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:48:04.572 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:48:04.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:48:04.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:48:04.574 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:48:04.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:04.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:48:04.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:48:04.575 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:04.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:48:04.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:48:04.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:48:04.581 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:48:04.581 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:04.581 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:04.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:48:04.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:48:04.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:48:04.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2026-02-22T21:48:04.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2026-02-22T21:48:04.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:04.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2026-02-22T21:48:04.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:04.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:48:04.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:48:04.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:48:04.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:48:04.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:48:04.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:48:04.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:48:04.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:48:04.586 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:48:04.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:48:04.587 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:48:04.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:48:04.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:48:04.588 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:48:04.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:04.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:48:04.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:48:04.590 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:04.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:48:04.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:48:04.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:48:04.593 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:48:04.593 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:04.593 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:04.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:48:04.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:48:04.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:48:04.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2026-02-22T21:48:04.595 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:48:04.595 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:04.595 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:04.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12473 2026-02-22T21:48:04.597 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:48:04.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:48:04.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:48:04.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-22T21:48:04.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2026-02-22T21:48:04.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:04.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:48:04.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:48:04.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:48:04.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-22T21:48:04.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2026-02-22T21:48:04.618 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:48:04.618 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:04.618 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:04.619 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:04.619 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:04.619 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:04.619 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:04.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2026-02-22T21:48:04.646 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:48:04.647 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:48:04.647 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:48:04.647 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:48:04.647 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:48:04.648 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:48:04.649 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:48:04.649 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:04.649 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:04.649 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:04.649 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:04.649 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:04.649 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:04.649 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:48:04.649 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get fsid 2026-02-22T21:48:04.703 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:48:04.704 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:48:04.704 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:48:04.704 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:48:04.704 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:48:04.704 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:48:04.704 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:04.704 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:04.705 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:04.705 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:04.706 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:04.706 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:04.706 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:48:04.706 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get mon_host 2026-02-22T21:48:04.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2026-02-22T21:48:04.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:04.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:48:04.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:48:04.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:48:04.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-22T21:48:04.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:48:04.862 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:48:04.862 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:04.862 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:04.862 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:04.862 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:04.862 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:04.863 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:04.863 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:48:04.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:48:04.883 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2026-02-22T21:48:04.884 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:04.884 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:04.884 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:04.884 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:04.884 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:04.884 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:04.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2026-02-22T21:48:04.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:48:04.933 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:48:04.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2026-02-22T21:48:04.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:04.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:48:04.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:48:04.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:48:04.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-22T21:48:04.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:48:04.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:48:04.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:48:04.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:48:04.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-22T21:48:04.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-22T21:48:04.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:48:04.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:48:04.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:48:04.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:48:04.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:48:04.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:48:04.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:48:04.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:48:04.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:48:04.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2026-02-22T21:48:04.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-22T21:48:04.945 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:48:04.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4a188a14-3f5d-4c5e-b88a-5a2c97222546 2026-02-22T21:48:04.946 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 4a188a14-3f5d-4c5e-b88a-5a2c97222546 2026-02-22T21:48:04.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4a188a14-3f5d-4c5e-b88a-5a2c97222546' 2026-02-22T21:48:04.947 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:48:04.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCUeZtpdRE5OBAAkVkUuKr6pbLUoi1Glj1JCA== 2026-02-22T21:48:04.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCUeZtpdRE5OBAAkVkUuKr6pbLUoi1Glj1JCA=="}' 2026-02-22T21:48:04.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4a188a14-3f5d-4c5e-b88a-5a2c97222546 -i td/test-erasure-code-plugins/0/new.json 2026-02-22T21:48:05.065 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:48:05.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-22T21:48:05.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQCUeZtpdRE5OBAAkVkUuKr6pbLUoi1Glj1JCA== --osd-uuid 4a188a14-3f5d-4c5e-b88a-5a2c97222546 2026-02-22T21:48:05.090 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:05.072+0000 7f36f2a9a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:05.091 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:05.076+0000 7f36f2a9a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:05.092 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:05.076+0000 7f36f2a9a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:05.093 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:05.076+0000 7f36f2a9a8c0 -1 bdev(0x562a38c33c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:48:05.093 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:05.076+0000 7f36f2a9a8c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-22T21:48:07.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-22T21:48:07.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:48:07.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:48:07.464 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:48:07.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:48:07.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:48:07.731 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:48:07.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2026-02-22T21:48:07.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:48:07.732 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:48:07.732 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:48:07.751 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:07.736+0000 7f357d1ea8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:07.753 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:07.736+0000 7f357d1ea8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:07.754 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:07.740+0000 7f357d1ea8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:07.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:48:07.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:48:07.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:48:07.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:48:07.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:48:07.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:07.930 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:48:07.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:48:07.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:07.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:08.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:09.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:09.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:09.120 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:48:09.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:48:09.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:09.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:09.256 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:09.240+0000 7f357d1ea8c0 -1 Falling back to public interface 2026-02-22T21:48:09.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:10.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:10.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:10.316 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:48:10.317 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:48:10.317 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:10.317 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:10.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:11.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:11.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:11.506 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:48:11.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:48:11.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:11.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:11.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:12.250 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:12.232+0000 7f357d1ea8c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:48:12.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:12.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:12.707 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:48:12.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:48:12.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:12.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:12.903 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/197862405,v1:127.0.0.1:6803/197862405] [v2:127.0.0.1:6804/197862405,v1:127.0.0.1:6805/197862405] exists,up 4a188a14-3f5d-4c5e-b88a-5a2c97222546 2026-02-22T21:48:12.903 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:48:12.903 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:48:12.903 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:48:12.904 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2026-02-22T21:48:12.904 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:48:12.904 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:48:12.904 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:12.904 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:12.904 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:12.904 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-osd.0.asok 2026-02-22T21:48:12.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2026-02-22T21:48:12.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-osd.0.asok log flush 2026-02-22T21:48:12.967 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:48:12.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:48:12.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2026-02-22T21:48:12.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2026-02-22T21:48:12.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:12.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:48:12.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:48:12.977 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:48:12.977 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:48:12.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:48:12.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:48:12.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:48:13.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:48:13.086 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:48:13.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:48:13.087 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:48:13.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:48:13.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:48:13.089 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:48:13.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:13.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:48:13.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:48:13.091 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:13.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:48:13.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:48:13.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:48:13.097 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:48:13.097 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:13.097 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:13.097 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:48:13.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:48:13.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:48:13.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2026-02-22T21:48:13.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2026-02-22T21:48:13.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:13.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2026-02-22T21:48:13.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:13.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:48:13.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:48:13.100 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:48:13.100 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:48:13.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:48:13.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:48:13.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:48:13.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:48:13.102 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:48:13.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:48:13.103 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:48:13.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:48:13.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:48:13.105 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:48:13.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:13.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:48:13.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:48:13.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:13.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:48:13.108 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:48:13.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:48:13.109 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:48:13.110 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:13.110 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:13.110 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:48:13.111 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:48:13.111 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:48:13.111 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2026-02-22T21:48:13.112 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:48:13.112 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:13.112 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:13.112 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12473 2026-02-22T21:48:13.114 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:48:13.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:48:13.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:48:13.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-22T21:48:13.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2026-02-22T21:48:13.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:13.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:48:13.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:48:13.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:48:13.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-22T21:48:13.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2026-02-22T21:48:13.136 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:48:13.136 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:13.136 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:13.136 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:13.136 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:13.136 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:13.136 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:13.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2026-02-22T21:48:13.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:48:13.161 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:48:13.161 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:48:13.161 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:48:13.162 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:48:13.162 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:48:13.162 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:48:13.162 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:13.162 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:13.163 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:13.163 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:13.163 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:13.163 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:13.163 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:48:13.163 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get fsid 2026-02-22T21:48:13.218 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:48:13.218 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:48:13.219 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:48:13.219 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:48:13.219 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:48:13.219 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:48:13.219 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:13.219 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:13.220 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:13.220 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:13.220 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:13.220 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:13.221 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:48:13.221 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get mon_host 2026-02-22T21:48:13.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2026-02-22T21:48:13.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:13.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:48:13.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:48:13.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:48:13.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-22T21:48:13.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:48:13.385 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:48:13.385 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:13.385 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:13.386 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:13.386 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:13.386 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:13.386 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:13.386 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:48:13.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:48:13.404 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2026-02-22T21:48:13.404 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:13.404 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:13.405 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:13.405 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:13.405 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:13.405 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:13.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2026-02-22T21:48:13.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:48:13.453 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:48:13.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2026-02-22T21:48:13.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:13.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:48:13.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:48:13.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:48:13.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-22T21:48:13.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:48:13.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:48:13.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:48:13.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:48:13.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-22T21:48:13.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-22T21:48:13.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:48:13.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:48:13.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:48:13.463 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:48:13.463 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:13.463 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:13.463 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:13.463 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:13.463 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:13.464 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:13.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:13.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:48:13.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:48:13.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:48:13.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:48:13.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:48:13.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:48:13.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:48:13.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:48:13.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:48:13.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:48:13.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2026-02-22T21:48:13.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-22T21:48:13.465 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:48:13.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0fdff733-d1bd-4d16-bf19-cb5466d88bc9 2026-02-22T21:48:13.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0fdff733-d1bd-4d16-bf19-cb5466d88bc9' 2026-02-22T21:48:13.466 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 0fdff733-d1bd-4d16-bf19-cb5466d88bc9 2026-02-22T21:48:13.466 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:48:13.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdeZtpBWCfGxAAiRziAoPBI0UIZ+hkG4z2/w== 2026-02-22T21:48:13.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdeZtpBWCfGxAAiRziAoPBI0UIZ+hkG4z2/w=="}' 2026-02-22T21:48:13.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0fdff733-d1bd-4d16-bf19-cb5466d88bc9 -i td/test-erasure-code-plugins/0/new.json 2026-02-22T21:48:13.583 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:48:13.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-22T21:48:13.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQCdeZtpBWCfGxAAiRziAoPBI0UIZ+hkG4z2/w== --osd-uuid 0fdff733-d1bd-4d16-bf19-cb5466d88bc9 2026-02-22T21:48:13.607 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:13.592+0000 7fbb0edc38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:13.608 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:13.592+0000 7fbb0edc38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:13.609 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:13.592+0000 7fbb0edc38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:13.610 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:13.592+0000 7fbb0edc38c0 -1 bdev(0x5648508bdc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:48:13.610 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:13.592+0000 7fbb0edc38c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-22T21:48:16.154 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-22T21:48:16.155 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:48:16.155 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:48:16.156 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:48:16.156 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:48:16.422 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:48:16.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:48:16.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2026-02-22T21:48:16.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:48:16.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:48:16.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:48:16.443 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:16.428+0000 7fb32cda08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:16.445 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:16.428+0000 7fb32cda08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:16.446 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:16.432+0000 7fb32cda08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:16.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:48:16.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:48:16.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:48:16.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:48:16.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:48:16.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:16.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:48:16.617 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:48:16.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:16.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:16.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:17.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:17.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:17.818 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:48:17.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:48:17.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:17.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:17.971 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:17.956+0000 7fb32cda08c0 -1 Falling back to public interface 2026-02-22T21:48:18.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:19.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:19.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:19.018 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:48:19.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:48:19.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:19.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:19.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:20.218 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:20.218 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:20.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:48:20.219 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:48:20.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:20.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:20.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:20.956 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:20.940+0000 7fb32cda08c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:48:21.414 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:21.414 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:21.414 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:48:21.415 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:48:21.415 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:21.415 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:21.613 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2328455826,v1:127.0.0.1:6803/2328455826] [v2:127.0.0.1:6804/2328455826,v1:127.0.0.1:6805/2328455826] exists,up 0fdff733-d1bd-4d16-bf19-cb5466d88bc9 2026-02-22T21:48:21.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:48:21.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:48:21.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:48:21.615 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2026-02-22T21:48:21.615 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:48:21.615 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:48:21.615 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:21.615 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:21.616 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:21.616 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-osd.0.asok 2026-02-22T21:48:21.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2026-02-22T21:48:21.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-osd.0.asok log flush 2026-02-22T21:48:21.677 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:48:21.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:48:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2026-02-22T21:48:21.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2026-02-22T21:48:21.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:21.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:48:21.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:48:21.687 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:48:21.687 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:48:21.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:48:21.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:48:21.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:48:21.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:48:21.795 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:48:21.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:48:21.796 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:48:21.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:48:21.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:48:21.798 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:48:21.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:21.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:48:21.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:48:21.799 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:21.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:48:21.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:48:21.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:48:21.806 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:48:21.806 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:21.806 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:21.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:48:21.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:48:21.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:48:21.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2026-02-22T21:48:21.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-22T21:48:21.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2026-02-22T21:48:21.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:21.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2026-02-22T21:48:21.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:21.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2026-02-22T21:48:21.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:21.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:48:21.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:48:21.810 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:48:21.810 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:48:21.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:48:21.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:48:21.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:48:21.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:48:21.811 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:48:21.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:48:21.811 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:48:21.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:48:21.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:48:21.813 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:48:21.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:21.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:48:21.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:48:21.815 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:21.815 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:48:21.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:48:21.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:48:21.818 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:48:21.818 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:21.818 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:21.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:48:21.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:48:21.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:48:21.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2026-02-22T21:48:21.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:48:21.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:21.821 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:21.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12473 2026-02-22T21:48:21.822 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:48:21.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:48:21.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:48:21.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-22T21:48:21.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2026-02-22T21:48:21.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:21.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:48:21.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:48:21.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:48:21.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-22T21:48:21.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2026-02-22T21:48:21.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:48:21.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:21.844 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:21.844 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:21.844 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:21.844 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:21.844 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:21.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T21:48:21.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:48:21.875 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:48:21.875 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:48:21.875 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:48:21.875 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:48:21.875 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:48:21.875 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:48:21.875 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:21.875 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:21.875 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:21.875 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:21.876 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:21.876 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:21.876 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:48:21.876 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get fsid 2026-02-22T21:48:21.930 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:48:21.930 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:48:21.930 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:48:21.930 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:48:21.930 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:48:21.930 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:48:21.930 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:21.931 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:21.931 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:21.932 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:21.932 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:21.932 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:21.932 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:48:21.932 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get mon_host 2026-02-22T21:48:21.988 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2026-02-22T21:48:21.989 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:21.989 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:21.989 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:21.989 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:21.989 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:21.989 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:21.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2026-02-22T21:48:21.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:48:22.045 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:48:22.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2026-02-22T21:48:22.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:22.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:48:22.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:48:22.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:48:22.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-22T21:48:22.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:48:22.172 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:48:22.172 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:22.172 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:22.172 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:22.172 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:22.172 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:22.172 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:22.173 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:48:22.173 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:48:22.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2026-02-22T21:48:22.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:22.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:48:22.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:48:22.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:48:22.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-22T21:48:22.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:48:22.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:48:22.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:48:22.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:48:22.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-22T21:48:22.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-22T21:48:22.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:48:22.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:48:22.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:48:22.194 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:48:22.194 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:22.194 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:22.194 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:22.194 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:48:22.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:48:22.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:48:22.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-22T21:48:22.196 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:48:22.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4ce22e12-7949-49cd-8d02-50fe1f90850b 2026-02-22T21:48:22.197 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 4ce22e12-7949-49cd-8d02-50fe1f90850b 2026-02-22T21:48:22.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4ce22e12-7949-49cd-8d02-50fe1f90850b' 2026-02-22T21:48:22.197 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:48:22.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCmeZtpVV2gCxAANKpoltFEcndQEplQ3rC4RQ== 2026-02-22T21:48:22.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCmeZtpVV2gCxAANKpoltFEcndQEplQ3rC4RQ=="}' 2026-02-22T21:48:22.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4ce22e12-7949-49cd-8d02-50fe1f90850b -i td/test-erasure-code-plugins/0/new.json 2026-02-22T21:48:22.313 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:48:22.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-22T21:48:22.322 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmeZtpVV2gCxAANKpoltFEcndQEplQ3rC4RQ== --osd-uuid 4ce22e12-7949-49cd-8d02-50fe1f90850b 2026-02-22T21:48:22.337 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:22.320+0000 7f6bd912d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:22.338 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:22.324+0000 7f6bd912d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:22.339 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:22.324+0000 7f6bd912d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:22.340 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:22.324+0000 7f6bd912d8c0 -1 bdev(0x555826e43c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:48:22.340 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:22.324+0000 7f6bd912d8c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-22T21:48:24.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-22T21:48:24.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:48:24.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:48:24.920 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:48:24.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:48:25.194 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:48:25.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:48:25.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:48:25.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:48:25.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:48:25.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:48:25.217 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:25.200+0000 7f4364e5f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:25.218 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:25.204+0000 7f4364e5f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:25.219 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:25.204+0000 7f4364e5f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:25.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:48:25.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:48:25.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:48:25.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:48:25.389 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:48:25.389 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:25.389 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:48:25.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:48:25.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:25.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:25.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:26.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:26.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:26.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:48:26.589 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:48:26.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:26.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:26.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:26.948 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:26.932+0000 7f4364e5f8c0 -1 Falling back to public interface 2026-02-22T21:48:27.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:27.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:27.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:48:27.786 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:48:27.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:27.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:27.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:28.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:28.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:28.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:48:28.983 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:48:28.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:28.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:29.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:29.938 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:29.924+0000 7f4364e5f8c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:48:30.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:30.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:30.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:48:30.182 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:48:30.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:30.183 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:30.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:31.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:31.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:31.381 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:48:31.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:48:31.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:31.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:31.578 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2691064412,v1:127.0.0.1:6803/2691064412] [v2:127.0.0.1:6804/2691064412,v1:127.0.0.1:6805/2691064412] exists,up 4ce22e12-7949-49cd-8d02-50fe1f90850b 2026-02-22T21:48:31.579 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:48:31.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:48:31.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:48:31.580 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2026-02-22T21:48:31.580 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:48:31.580 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:48:31.580 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:31.580 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:31.580 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:31.581 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-osd.0.asok 2026-02-22T21:48:31.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2026-02-22T21:48:31.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-osd.0.asok log flush 2026-02-22T21:48:31.643 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:48:31.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:48:31.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2026-02-22T21:48:31.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2026-02-22T21:48:31.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:31.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:48:31.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:48:31.653 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:48:31.654 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:48:31.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:48:31.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:48:31.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:48:31.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:48:31.761 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:48:31.762 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:48:31.762 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:48:31.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:48:31.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:48:31.764 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:48:31.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:31.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:48:31.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:48:31.766 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:31.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:48:31.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:48:31.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:48:31.771 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:48:31.771 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:31.771 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:31.771 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:48:31.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:48:31.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:48:31.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2026-02-22T21:48:31.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2026-02-22T21:48:31.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2026-02-22T21:48:31.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:31.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-22T21:48:31.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-22T21:48:31.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:31.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2026-02-22T21:48:31.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:31.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:48:31.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:48:31.775 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:48:31.775 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:48:31.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:48:31.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:48:31.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:48:31.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:48:31.776 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:48:31.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:48:31.777 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:48:31.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:48:31.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:48:31.778 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:48:31.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:31.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:48:31.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:48:31.780 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:31.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:48:31.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:48:31.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:48:31.784 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:48:31.784 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:31.784 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:31.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:48:31.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:48:31.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:48:31.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2026-02-22T21:48:31.787 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:48:31.787 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:31.787 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:31.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12473 2026-02-22T21:48:31.788 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:48:31.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:48:31.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:48:31.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-22T21:48:31.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2026-02-22T21:48:31.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:31.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:48:31.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:48:31.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:48:31.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-22T21:48:31.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2026-02-22T21:48:31.809 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:48:31.809 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:31.809 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:31.809 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:31.809 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:31.809 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:31.809 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:31.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2026-02-22T21:48:31.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:48:31.838 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:48:31.838 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:48:31.839 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:48:31.839 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:48:31.839 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:48:31.840 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:48:31.840 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:31.840 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:31.840 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:31.840 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:31.840 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:31.841 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:31.841 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:48:31.841 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get fsid 2026-02-22T21:48:31.896 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:48:31.896 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:48:31.896 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:48:31.897 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:48:31.897 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:48:31.897 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:48:31.897 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:31.897 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:31.898 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:31.898 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:31.898 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:31.898 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:31.898 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:48:31.898 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get mon_host 2026-02-22T21:48:31.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-22T21:48:31.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:31.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:48:31.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:48:31.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:48:31.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-22T21:48:31.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:48:32.059 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:48:32.059 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:32.059 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:32.059 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:32.059 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:32.059 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:32.059 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:32.060 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:48:32.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:48:32.080 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-22T21:48:32.080 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:32.080 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:32.080 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:32.080 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:32.080 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:32.080 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:32.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-22T21:48:32.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:48:32.128 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:48:32.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2026-02-22T21:48:32.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:32.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:48:32.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:48:32.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:48:32.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-22T21:48:32.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:48:32.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:48:32.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:48:32.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:48:32.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-22T21:48:32.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-22T21:48:32.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:48:32.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:48:32.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:48:32.138 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:48:32.138 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:32.138 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:32.138 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:32.138 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:32.138 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:32.138 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:32.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:32.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:48:32.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:48:32.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:48:32.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:48:32.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:48:32.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:48:32.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:48:32.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:48:32.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:48:32.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:48:32.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2026-02-22T21:48:32.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-22T21:48:32.140 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:48:32.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f224a903-5be6-4911-aa04-bc72faf17beb 2026-02-22T21:48:32.141 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 f224a903-5be6-4911-aa04-bc72faf17beb 2026-02-22T21:48:32.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f224a903-5be6-4911-aa04-bc72faf17beb' 2026-02-22T21:48:32.141 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:48:32.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCweZtpqjZGCBAAZkQbrQcfI9QIRzNDlKztfw== 2026-02-22T21:48:32.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCweZtpqjZGCBAAZkQbrQcfI9QIRzNDlKztfw=="}' 2026-02-22T21:48:32.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f224a903-5be6-4911-aa04-bc72faf17beb -i td/test-erasure-code-plugins/0/new.json 2026-02-22T21:48:32.256 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:48:32.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-22T21:48:32.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQCweZtpqjZGCBAAZkQbrQcfI9QIRzNDlKztfw== --osd-uuid f224a903-5be6-4911-aa04-bc72faf17beb 2026-02-22T21:48:32.279 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:32.268+0000 7fcd8e6958c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:32.280 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:32.268+0000 7fcd8e6958c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:32.281 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:32.268+0000 7fcd8e6958c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:32.282 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:32.272+0000 7fcd8e6958c0 -1 bdev(0x561f4f491c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:48:32.283 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:32.272+0000 7fcd8e6958c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-22T21:48:34.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-22T21:48:34.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:48:34.843 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:48:34.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:48:34.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:48:35.113 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:48:35.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:48:35.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2026-02-22T21:48:35.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:48:35.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:48:35.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:48:35.136 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:35.124+0000 7ff31afaa8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:35.137 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:35.124+0000 7ff31afaa8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:35.138 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:35.128+0000 7ff31afaa8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:35.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:48:35.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:48:35.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:48:35.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:48:35.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:48:35.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:35.309 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:48:35.309 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:48:35.309 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:35.309 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:35.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:36.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:36.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:36.504 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:48:36.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:48:36.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:36.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:36.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:36.896 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:36.884+0000 7ff31afaa8c0 -1 Falling back to public interface 2026-02-22T21:48:37.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:37.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:37.706 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:48:37.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:48:37.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:37.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:37.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:38.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:38.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:38.908 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:48:38.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:48:38.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:38.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:39.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:39.953 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:39.940+0000 7ff31afaa8c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:48:40.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:40.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:40.104 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:48:40.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:48:40.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:40.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:40.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:41.307 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:41.307 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:41.308 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:48:41.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:48:41.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:41.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:41.509 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1976507039,v1:127.0.0.1:6803/1976507039] [v2:127.0.0.1:6804/1976507039,v1:127.0.0.1:6805/1976507039] exists,up f224a903-5be6-4911-aa04-bc72faf17beb 2026-02-22T21:48:41.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:48:41.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:48:41.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:48:41.511 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-22T21:48:41.511 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:48:41.511 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:48:41.511 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:41.511 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:41.511 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:41.511 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-osd.0.asok 2026-02-22T21:48:41.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-22T21:48:41.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-osd.0.asok log flush 2026-02-22T21:48:41.572 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:48:41.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:48:41.579 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:48:31.816+0000 7fbf32a37ec0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-22T21:48:41.579 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2026-02-22T21:48:41.581 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:48:36.884+0000 7ff31afaa8c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-22T21:48:41.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-22T21:48:41.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:41.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:48:41.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:48:41.583 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:48:41.583 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:48:41.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:48:41.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:48:41.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:48:41.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:48:41.691 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:48:41.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:48:41.692 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:48:41.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:48:41.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:48:41.694 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:48:41.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:41.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:48:41.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:48:41.696 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:41.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:48:41.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:48:41.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:48:41.701 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:48:41.701 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:41.701 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:41.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:48:41.703 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:48:41.703 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:48:41.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-22T21:48:41.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-22T21:48:41.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:41.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2026-02-22T21:48:41.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:41.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:48:41.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:48:41.704 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:48:41.704 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:48:41.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:48:41.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:48:41.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:48:41.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:48:41.706 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:48:41.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:48:41.707 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:48:41.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:48:41.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:48:41.708 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:48:41.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:41.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:48:41.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:48:41.710 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:41.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:48:41.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:48:41.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:48:41.713 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:48:41.713 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:41.714 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:41.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:48:41.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:48:41.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:48:41.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2026-02-22T21:48:41.716 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:48:41.716 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:41.716 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:41.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12473 2026-02-22T21:48:41.718 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:48:41.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:48:41.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:48:41.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-22T21:48:41.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2026-02-22T21:48:41.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:48:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:48:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:48:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-22T21:48:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2026-02-22T21:48:41.739 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:48:41.739 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:41.740 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:41.740 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:41.740 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:41.740 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:41.740 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:41.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2026-02-22T21:48:41.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:48:41.769 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:48:41.769 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:48:41.769 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:48:41.769 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:48:41.770 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:48:41.771 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:48:41.771 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:41.771 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:41.771 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:41.771 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:41.771 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:41.771 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:41.772 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:48:41.772 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get fsid 2026-02-22T21:48:41.826 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:48:41.827 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:48:41.827 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:48:41.827 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:48:41.827 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:48:41.828 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:48:41.829 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:41.829 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:41.829 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:41.829 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:41.829 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:41.829 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:41.829 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:48:41.829 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get mon_host 2026-02-22T21:48:41.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-22T21:48:41.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:41.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:48:41.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:48:41.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:48:41.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-22T21:48:41.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:48:41.994 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:48:41.994 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:41.994 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:41.994 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:41.994 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:41.994 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:41.994 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:41.995 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:48:41.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:48:42.012 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-22T21:48:42.013 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:42.013 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:42.013 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:42.013 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:42.013 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:42.013 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:42.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-22T21:48:42.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:48:42.061 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:48:42.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2026-02-22T21:48:42.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:42.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:48:42.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:48:42.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:48:42.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-22T21:48:42.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:48:42.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:48:42.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:48:42.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:48:42.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-22T21:48:42.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-22T21:48:42.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:48:42.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:48:42.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:48:42.072 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:48:42.072 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:42.072 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:42.072 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:42.072 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:42.072 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:42.072 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:42.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:42.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:48:42.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:48:42.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:48:42.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:48:42.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:48:42.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:48:42.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:48:42.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:48:42.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:48:42.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:48:42.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2026-02-22T21:48:42.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-22T21:48:42.073 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:48:42.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=90181788-5cc4-4795-996b-31e57a70b4ca 2026-02-22T21:48:42.074 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 90181788-5cc4-4795-996b-31e57a70b4ca 2026-02-22T21:48:42.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 90181788-5cc4-4795-996b-31e57a70b4ca' 2026-02-22T21:48:42.075 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:48:42.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6eZtpRKAtBBAABTF0R0nOf3jl8xquA0x6sg== 2026-02-22T21:48:42.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6eZtpRKAtBBAABTF0R0nOf3jl8xquA0x6sg=="}' 2026-02-22T21:48:42.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 90181788-5cc4-4795-996b-31e57a70b4ca -i td/test-erasure-code-plugins/0/new.json 2026-02-22T21:48:42.188 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:48:42.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-22T21:48:42.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQC6eZtpRKAtBBAABTF0R0nOf3jl8xquA0x6sg== --osd-uuid 90181788-5cc4-4795-996b-31e57a70b4ca 2026-02-22T21:48:42.211 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:42.200+0000 7f0119cb08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:42.212 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:42.200+0000 7f0119cb08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:42.213 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:42.200+0000 7f0119cb08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:42.214 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:42.200+0000 7f0119cb08c0 -1 bdev(0x56274cfffc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:48:42.214 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:42.204+0000 7f0119cb08c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-22T21:48:44.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-22T21:48:44.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:48:44.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:48:44.825 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:48:44.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:48:45.094 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:48:45.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:48:45.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2026-02-22T21:48:45.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:48:45.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:48:45.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:48:45.116 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:45.104+0000 7f912f6a88c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:45.118 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:45.108+0000 7f912f6a88c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:45.119 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:45.108+0000 7f912f6a88c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:45.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:48:45.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:48:45.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:48:45.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:48:45.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:48:45.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:45.286 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:48:45.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:48:45.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:45.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:45.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:46.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:46.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:46.487 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:48:46.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:48:46.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:46.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:46.595 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:46.584+0000 7f912f6a88c0 -1 Falling back to public interface 2026-02-22T21:48:46.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:47.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:47.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:47.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:48:47.689 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:48:47.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:47.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:47.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:48.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:48.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:48.888 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:48:48.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:48:48.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:48.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:49.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:49.806 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:49.792+0000 7f912f6a88c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:48:50.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:50.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:50.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:48:50.088 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:48:50.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:50.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:50.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:51.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:51.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:51.292 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:48:51.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:48:51.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:51.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:51.486 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4103370455,v1:127.0.0.1:6803/4103370455] [v2:127.0.0.1:6804/4103370455,v1:127.0.0.1:6805/4103370455] exists,up 90181788-5cc4-4795-996b-31e57a70b4ca 2026-02-22T21:48:51.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:48:51.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:48:51.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:48:51.488 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-22T21:48:51.488 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:48:51.489 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:48:51.489 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:51.489 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:51.489 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:51.489 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-osd.0.asok 2026-02-22T21:48:51.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-22T21:48:51.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-osd.0.asok log flush 2026-02-22T21:48:51.543 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:48:51.549 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:48:51.550 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:48:41.744+0000 7f212cd43ec0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-22T21:48:51.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2026-02-22T21:48:51.552 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:48:46.584+0000 7f912f6a88c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-22T21:48:51.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-22T21:48:51.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:51.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:48:51.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:48:51.554 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:48:51.554 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:48:51.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:48:51.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:48:51.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:48:51.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:48:51.662 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:48:51.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:48:51.663 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:48:51.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:48:51.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:48:51.665 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:48:51.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:51.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:48:51.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:48:51.667 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:51.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:48:51.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:48:51.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:48:51.673 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:48:51.673 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:51.673 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:51.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:48:51.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:48:51.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:48:51.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-22T21:48:51.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-22T21:48:51.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:51.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2026-02-22T21:48:51.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:51.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:48:51.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:48:51.676 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:48:51.676 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:48:51.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:48:51.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:48:51.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:48:51.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:48:51.678 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:48:51.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:48:51.679 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:48:51.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:48:51.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:48:51.680 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:48:51.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:51.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:48:51.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:48:51.682 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:48:51.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:48:51.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:48:51.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:48:51.685 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:48:51.685 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:51.685 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:51.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:48:51.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:48:51.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:48:51.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2026-02-22T21:48:51.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:48:51.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:51.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:51.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12473 2026-02-22T21:48:51.689 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:48:51.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:48:51.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:48:51.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-22T21:48:51.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2026-02-22T21:48:51.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:51.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:48:51.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:48:51.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:48:51.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-22T21:48:51.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2026-02-22T21:48:51.711 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:48:51.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:51.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:51.712 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:51.712 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:51.712 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:51.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:51.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2026-02-22T21:48:51.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:48:51.739 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:48:51.739 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:48:51.739 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:48:51.739 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:48:51.739 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:48:51.741 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:48:51.741 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:51.741 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:51.741 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:51.741 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:51.741 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:51.742 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:51.742 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:48:51.742 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get fsid 2026-02-22T21:48:51.796 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:48:51.797 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:48:51.797 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:48:51.797 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:48:51.797 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:48:51.797 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:48:51.797 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:51.797 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:51.798 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:51.798 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:51.799 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:51.799 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:51.799 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:48:51.799 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get mon_host 2026-02-22T21:48:51.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-22T21:48:51.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:51.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:48:51.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:48:51.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:48:51.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-22T21:48:51.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:48:51.958 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:48:51.958 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:51.958 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:51.959 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:51.959 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:51.959 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:51.959 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:51.959 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:48:51.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:48:51.978 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-22T21:48:51.979 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:48:51.979 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:48:51.979 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:48:51.979 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:51.979 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:51.979 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:48:51.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-22T21:48:51.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:48:52.028 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:48:52.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2026-02-22T21:48:52.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:48:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:48:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:48:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:48:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-22T21:48:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:48:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:48:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:48:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:48:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-22T21:48:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-22T21:48:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:48:52.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:48:52.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:48:52.039 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:48:52.039 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:48:52.039 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:48:52.039 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:48:52.039 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:48:52.039 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:48:52.039 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:52.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:48:52.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:48:52.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:48:52.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:48:52.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:48:52.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:48:52.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:48:52.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:48:52.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:48:52.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:48:52.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:48:52.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2026-02-22T21:48:52.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-22T21:48:52.041 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:48:52.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c7638dd4-3936-4393-8a48-ce568e705e0d 2026-02-22T21:48:52.041 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 c7638dd4-3936-4393-8a48-ce568e705e0d 2026-02-22T21:48:52.042 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c7638dd4-3936-4393-8a48-ce568e705e0d' 2026-02-22T21:48:52.042 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:48:52.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDEeZtp5/NgAhAAtCkWJpTnnJvnB7mgddt3GA== 2026-02-22T21:48:52.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDEeZtp5/NgAhAAtCkWJpTnnJvnB7mgddt3GA=="}' 2026-02-22T21:48:52.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c7638dd4-3936-4393-8a48-ce568e705e0d -i td/test-erasure-code-plugins/0/new.json 2026-02-22T21:48:52.159 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:48:52.167 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-22T21:48:52.168 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQDEeZtp5/NgAhAAtCkWJpTnnJvnB7mgddt3GA== --osd-uuid c7638dd4-3936-4393-8a48-ce568e705e0d 2026-02-22T21:48:52.183 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:52.171+0000 7f1e90ba78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:52.185 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:52.171+0000 7f1e90ba78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:52.186 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:52.175+0000 7f1e90ba78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:52.186 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:52.175+0000 7f1e90ba78c0 -1 bdev(0x5569403d7c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:48:52.186 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:52.175+0000 7f1e90ba78c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-22T21:48:54.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-22T21:48:54.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:48:54.506 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:48:54.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:48:54.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:48:54.763 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:48:54.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:48:54.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2026-02-22T21:48:54.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:48:54.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:48:54.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:48:54.785 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:54.771+0000 7f46880c28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:54.787 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:54.775+0000 7f46880c28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:54.788 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:54.775+0000 7f46880c28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:48:54.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:48:54.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:48:54.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:48:54.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:48:54.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:48:54.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:54.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:48:54.963 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:48:54.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:54.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:55.153 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:56.154 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:56.154 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:56.154 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:48:56.155 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:48:56.155 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:56.155 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:56.299 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:56.287+0000 7f46880c28c0 -1 Falling back to public interface 2026-02-22T21:48:56.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:57.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:57.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:57.349 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:48:57.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:48:57.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:57.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:57.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:58.542 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:58.542 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:58.542 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:48:58.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:48:58.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:58.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:58.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:48:59.518 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:48:59.507+0000 7f46880c28c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:48:59.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:48:59.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:48:59.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:48:59.742 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:48:59.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:48:59.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:48:59.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:00.941 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:00.941 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:00.942 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:49:00.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:49:00.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:00.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:01.138 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2322075022,v1:127.0.0.1:6803/2322075022] [v2:127.0.0.1:6804/2322075022,v1:127.0.0.1:6805/2322075022] exists,up c7638dd4-3936-4393-8a48-ce568e705e0d 2026-02-22T21:49:01.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:49:01.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:49:01.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:49:01.140 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-22T21:49:01.140 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:49:01.140 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:49:01.140 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:01.140 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:01.140 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:01.140 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-osd.0.asok 2026-02-22T21:49:01.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-22T21:49:01.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-osd.0.asok log flush 2026-02-22T21:49:01.201 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:49:01.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:49:01.208 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:48:51.716+0000 7fbd83abbec0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-22T21:49:01.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2026-02-22T21:49:01.209 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:48:56.287+0000 7f46880c28c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2026-02-22T21:49:01.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-22T21:49:01.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:01.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:49:01.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:49:01.211 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:49:01.211 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:49:01.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:49:01.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:49:01.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:49:01.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:49:01.319 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:49:01.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:49:01.320 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:49:01.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:49:01.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:49:01.322 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:49:01.322 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:01.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:49:01.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:49:01.323 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:01.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:49:01.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:49:01.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:49:01.330 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:49:01.330 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:01.331 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:01.331 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:49:01.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:49:01.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:49:01.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-22T21:49:01.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-22T21:49:01.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:01.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2026-02-22T21:49:01.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:01.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:49:01.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:49:01.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:49:01.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:49:01.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:49:01.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:49:01.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:49:01.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:49:01.335 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:49:01.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:49:01.336 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:49:01.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:49:01.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:49:01.338 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:49:01.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:01.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:49:01.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:49:01.339 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:01.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:49:01.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:49:01.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:49:01.343 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:49:01.343 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:01.343 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:01.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:49:01.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:49:01.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:49:01.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2026-02-22T21:49:01.346 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:49:01.346 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:01.346 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:01.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12473 2026-02-22T21:49:01.347 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:49:01.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:49:01.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:49:01.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-22T21:49:01.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2026-02-22T21:49:01.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:01.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:49:01.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:49:01.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:49:01.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-22T21:49:01.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2026-02-22T21:49:01.369 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:49:01.369 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:01.369 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:01.369 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:01.369 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:01.370 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:01.370 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:49:01.370 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2026-02-22T21:49:01.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:49:01.397 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:49:01.397 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:49:01.397 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:49:01.397 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:49:01.398 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:49:01.398 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:49:01.398 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:49:01.398 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:49:01.399 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:01.399 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:01.399 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:01.399 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:49:01.399 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:49:01.399 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get fsid 2026-02-22T21:49:01.453 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:49:01.453 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:49:01.454 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:49:01.454 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:49:01.454 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:49:01.454 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:49:01.454 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:49:01.454 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:49:01.455 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:01.455 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:01.455 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:01.455 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:49:01.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:49:01.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get mon_host 2026-02-22T21:49:01.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-22T21:49:01.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:01.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:49:01.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:49:01.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:49:01.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-22T21:49:01.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:49:01.614 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:49:01.615 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:01.615 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:01.615 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:01.615 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:01.615 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:01.615 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:49:01.615 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:49:01.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:49:01.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-22T21:49:01.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:49:01.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:49:01.635 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:01.635 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:01.635 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:01.636 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:49:01.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-22T21:49:01.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:49:01.685 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:49:01.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2026-02-22T21:49:01.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:01.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:49:01.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:49:01.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:49:01.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-22T21:49:01.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:49:01.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:49:01.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:49:01.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:49:01.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-22T21:49:01.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-22T21:49:01.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:49:01.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:49:01.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:49:01.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:49:01.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:49:01.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:49:01.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:49:01.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:49:01.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:49:01.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2026-02-22T21:49:01.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-22T21:49:01.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:49:01.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f774541d-3d35-4399-ae6a-318ff065f700 2026-02-22T21:49:01.697 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 f774541d-3d35-4399-ae6a-318ff065f700 2026-02-22T21:49:01.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f774541d-3d35-4399-ae6a-318ff065f700' 2026-02-22T21:49:01.698 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:49:01.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDNeZtp2Tl5KRAARnmTOvvKlK2RtChEcT8mzQ== 2026-02-22T21:49:01.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDNeZtp2Tl5KRAARnmTOvvKlK2RtChEcT8mzQ=="}' 2026-02-22T21:49:01.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f774541d-3d35-4399-ae6a-318ff065f700 -i td/test-erasure-code-plugins/0/new.json 2026-02-22T21:49:01.813 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:49:01.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-22T21:49:01.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQDNeZtp2Tl5KRAARnmTOvvKlK2RtChEcT8mzQ== --osd-uuid f774541d-3d35-4399-ae6a-318ff065f700 2026-02-22T21:49:01.838 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:01.827+0000 7f081b7cc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:01.839 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:01.827+0000 7f081b7cc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:01.840 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:01.827+0000 7f081b7cc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:01.841 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:01.827+0000 7f081b7cc8c0 -1 bdev(0x558c3fd29c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:49:01.841 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:01.827+0000 7f081b7cc8c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-22T21:49:04.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-22T21:49:04.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:49:04.691 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:49:04.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:49:04.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:49:04.954 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:49:04.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:49:04.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2026-02-22T21:49:04.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:49:04.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:49:04.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:49:04.976 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:04.963+0000 7f225ba038c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:04.977 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:04.967+0000 7f225ba038c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:04.979 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:04.967+0000 7f225ba038c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:05.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:49:05.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:49:05.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:49:05.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:49:05.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:49:05.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:05.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:49:05.161 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:49:05.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:05.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:05.352 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:06.353 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:06.353 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:06.354 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:49:06.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:49:06.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:06.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:06.491 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:06.479+0000 7f225ba038c0 -1 Falling back to public interface 2026-02-22T21:49:06.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:07.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:07.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:07.547 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:49:07.547 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:49:07.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:07.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:07.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:08.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:08.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:08.739 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:49:08.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:49:08.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:08.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:08.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:09.471 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:09.459+0000 7f225ba038c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:49:09.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:09.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:09.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:49:09.933 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:49:09.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:09.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:10.129 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/763414389,v1:127.0.0.1:6803/763414389] [v2:127.0.0.1:6804/763414389,v1:127.0.0.1:6805/763414389] exists,up f774541d-3d35-4399-ae6a-318ff065f700 2026-02-22T21:49:10.129 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:49:10.130 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:49:10.130 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:49:10.130 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-22T21:49:10.130 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:49:10.130 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:49:10.130 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:10.130 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:10.130 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:10.131 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-osd.0.asok 2026-02-22T21:49:10.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-22T21:49:10.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-osd.0.asok log flush 2026-02-22T21:49:10.193 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:49:10.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:49:10.200 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:49:01.375+0000 7f3ac6559ec0 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-22T21:49:10.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2026-02-22T21:49:10.201 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:49:06.479+0000 7f225ba038c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-22T21:49:10.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-22T21:49:10.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:10.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:49:10.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:49:10.202 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:49:10.202 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:49:10.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:49:10.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:49:10.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:49:10.310 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:49:10.310 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:49:10.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:49:10.311 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:49:10.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:49:10.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:49:10.313 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:49:10.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:10.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:49:10.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:49:10.315 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:10.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:49:10.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:49:10.317 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:49:10.321 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:49:10.321 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:10.322 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:10.322 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:49:10.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:49:10.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:49:10.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-22T21:49:10.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-22T21:49:10.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:10.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2026-02-22T21:49:10.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:10.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:49:10.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:49:10.325 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:49:10.325 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:49:10.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:49:10.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:49:10.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:49:10.326 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:49:10.326 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:49:10.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:49:10.327 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:49:10.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:49:10.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:49:10.329 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:49:10.330 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:10.330 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:49:10.330 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:49:10.330 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:10.331 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:49:10.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:49:10.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:49:10.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:49:10.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:10.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:10.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:49:10.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:49:10.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:49:10.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2026-02-22T21:49:10.336 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:49:10.336 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:10.337 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:10.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12473 2026-02-22T21:49:10.338 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:49:10.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:49:10.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:49:10.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-22T21:49:10.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2026-02-22T21:49:10.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:10.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:49:10.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:49:10.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:49:10.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-22T21:49:10.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2026-02-22T21:49:10.359 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:49:10.360 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:10.360 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:10.360 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:10.360 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:10.360 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:10.360 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:49:10.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2026-02-22T21:49:10.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:49:10.389 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:49:10.389 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:49:10.389 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:49:10.389 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:49:10.389 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:49:10.389 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:49:10.389 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:49:10.389 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:49:10.389 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:10.390 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:10.390 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:10.390 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:49:10.390 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:49:10.391 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get fsid 2026-02-22T21:49:10.445 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:49:10.445 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:49:10.446 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:49:10.446 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:49:10.446 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:49:10.446 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:49:10.446 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:49:10.446 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:49:10.447 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:10.447 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:10.447 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:10.447 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:49:10.447 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:49:10.447 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get mon_host 2026-02-22T21:49:10.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-22T21:49:10.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:10.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:49:10.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:49:10.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:49:10.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-22T21:49:10.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:49:10.607 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:49:10.608 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:10.608 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:10.608 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:10.608 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:10.608 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:10.608 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:49:10.609 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:49:10.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:49:10.627 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-22T21:49:10.627 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:49:10.628 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:49:10.628 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:10.628 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:10.628 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:10.628 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:49:10.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-22T21:49:10.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:49:10.677 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:49:10.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2026-02-22T21:49:10.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:10.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:49:10.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:49:10.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:49:10.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-22T21:49:10.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:49:10.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:49:10.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:49:10.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:49:10.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-22T21:49:10.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-22T21:49:10.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:49:10.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:49:10.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:49:10.687 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:49:10.687 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:10.687 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:10.688 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:10.688 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:10.688 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:10.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:49:10.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:49:10.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:49:10.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:49:10.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:49:10.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:49:10.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:49:10.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:49:10.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:49:10.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:49:10.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:49:10.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:49:10.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2026-02-22T21:49:10.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-22T21:49:10.690 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:49:10.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7c390c7b-a0d4-424b-baf2-901bc16b0ebe 2026-02-22T21:49:10.690 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 7c390c7b-a0d4-424b-baf2-901bc16b0ebe 2026-02-22T21:49:10.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7c390c7b-a0d4-424b-baf2-901bc16b0ebe' 2026-02-22T21:49:10.691 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:49:10.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWeZtpiKIFKRAAuFtvsmDWTlBKSsB/FLjImQ== 2026-02-22T21:49:10.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWeZtpiKIFKRAAuFtvsmDWTlBKSsB/FLjImQ=="}' 2026-02-22T21:49:10.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7c390c7b-a0d4-424b-baf2-901bc16b0ebe -i td/test-erasure-code-plugins/0/new.json 2026-02-22T21:49:10.806 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:49:10.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-22T21:49:10.815 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQDWeZtpiKIFKRAAuFtvsmDWTlBKSsB/FLjImQ== --osd-uuid 7c390c7b-a0d4-424b-baf2-901bc16b0ebe 2026-02-22T21:49:10.830 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:10.819+0000 7fbeff6bd8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:10.832 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:10.819+0000 7fbeff6bd8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:10.833 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:10.819+0000 7fbeff6bd8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:10.833 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:10.819+0000 7fbeff6bd8c0 -1 bdev(0x55a44c971c00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:49:10.833 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:10.819+0000 7fbeff6bd8c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-22T21:49:13.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-22T21:49:13.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:49:13.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:49:13.408 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:49:13.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:49:13.672 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:49:13.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:49:13.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2026-02-22T21:49:13.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:49:13.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:49:13.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:49:13.693 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:13.679+0000 7f63be9438c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:13.695 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:13.683+0000 7f63be9438c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:13.696 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:13.683+0000 7f63be9438c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:13.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:49:13.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:49:13.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:49:13.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:49:13.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:49:13.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:13.871 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:49:13.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:49:13.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:13.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:14.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:15.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:15.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:15.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:49:15.066 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:49:15.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:15.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:15.183 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:15.171+0000 7f63be9438c0 -1 Falling back to public interface 2026-02-22T21:49:15.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:16.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:16.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:16.263 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:49:16.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:49:16.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:16.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:16.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:17.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:17.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:17.456 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:49:17.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:49:17.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:17.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:17.647 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:18.211 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:18.199+0000 7f63be9438c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:49:18.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:18.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:18.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:49:18.649 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:49:18.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:18.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:18.848 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2261855097,v1:127.0.0.1:6803/2261855097] [v2:127.0.0.1:6804/2261855097,v1:127.0.0.1:6805/2261855097] exists,up 7c390c7b-a0d4-424b-baf2-901bc16b0ebe 2026-02-22T21:49:18.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:49:18.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:49:18.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:49:18.849 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-22T21:49:18.849 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:49:18.849 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:49:18.850 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:18.850 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:18.850 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:18.850 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-osd.0.asok 2026-02-22T21:49:18.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-22T21:49:18.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-osd.0.asok log flush 2026-02-22T21:49:18.904 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:49:18.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:49:18.911 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:49:10.363+0000 7f4d8f104ec0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-22T21:49:18.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2026-02-22T21:49:18.913 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:49:15.171+0000 7f63be9438c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-22T21:49:18.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-22T21:49:18.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:18.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:49:18.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:49:18.914 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:49:18.914 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:49:18.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:49:18.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:49:18.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:49:19.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:49:19.023 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:49:19.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:49:19.025 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:49:19.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:49:19.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:49:19.026 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:49:19.027 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:19.028 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:49:19.028 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:49:19.028 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:19.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:49:19.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:49:19.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:49:19.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:49:19.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:19.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:19.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:49:19.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:49:19.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:49:19.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2026-02-22T21:49:19.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2026-02-22T21:49:19.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:19.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2026-02-22T21:49:19.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:19.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:49:19.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:49:19.037 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:49:19.037 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:49:19.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:49:19.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:49:19.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:49:19.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:49:19.039 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:49:19.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:49:19.040 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:49:19.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:49:19.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:49:19.041 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:49:19.042 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:19.042 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:49:19.043 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:49:19.043 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:19.044 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:49:19.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:49:19.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:49:19.046 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:49:19.046 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:19.046 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:19.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:49:19.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:49:19.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:49:19.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2026-02-22T21:49:19.049 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:49:19.049 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:19.049 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:19.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.12473 2026-02-22T21:49:19.050 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:49:19.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:49:19.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:49:19.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2026-02-22T21:49:19.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2026-02-22T21:49:19.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:19.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:49:19.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:49:19.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:49:19.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2026-02-22T21:49:19.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2026-02-22T21:49:19.073 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:49:19.073 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:19.073 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:19.073 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:19.073 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:19.074 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:19.074 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:49:19.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2026-02-22T21:49:19.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:49:19.102 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:49:19.102 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:49:19.102 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:49:19.102 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:49:19.102 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:49:19.103 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:49:19.103 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:49:19.103 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:49:19.103 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:19.103 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:19.103 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:19.103 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:49:19.103 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:49:19.103 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get fsid 2026-02-22T21:49:19.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:49:19.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:49:19.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:49:19.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:49:19.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:49:19.158 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:49:19.158 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:49:19.159 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:49:19.159 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:19.159 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:19.159 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:19.159 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:49:19.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:49:19.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.12473/ceph-mon.a.asok config get mon_host 2026-02-22T21:49:19.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2026-02-22T21:49:19.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:19.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:49:19.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:49:19.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:49:19.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2026-02-22T21:49:19.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:49:19.319 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:49:19.319 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:19.319 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:19.319 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:19.319 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:19.319 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:19.319 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:49:19.320 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:49:19.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:49:19.339 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2026-02-22T21:49:19.339 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:49:19.339 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:49:19.339 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:19.340 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:19.340 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:19.340 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-mon.a.asok 2026-02-22T21:49:19.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2026-02-22T21:49:19.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-mon.a.asok log flush 2026-02-22T21:49:19.388 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:49:19.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2026-02-22T21:49:19.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:19.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:49:19.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:49:19.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:49:19.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2026-02-22T21:49:19.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 ' 2026-02-22T21:49:19.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:49:19.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:49:19.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:49:19.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2026-02-22T21:49:19.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2026-02-22T21:49:19.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:49:19.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:49:19.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2026-02-22T21:49:19.398 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:49:19.398 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:19.398 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:19.398 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:19.398 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:19.398 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:19.398 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2026-02-22T21:49:19.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2026-02-22T21:49:19.400 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:49:19.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5118f4d1-704f-4d21-a89e-104f3f7a8a4d 2026-02-22T21:49:19.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5118f4d1-704f-4d21-a89e-104f3f7a8a4d' 2026-02-22T21:49:19.401 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 5118f4d1-704f-4d21-a89e-104f3f7a8a4d 2026-02-22T21:49:19.401 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:49:19.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfeZtpP2/OFxAAG0jc8D8An7kw+/q06Wsv/w== 2026-02-22T21:49:19.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfeZtpP2/OFxAAG0jc8D8An7kw+/q06Wsv/w=="}' 2026-02-22T21:49:19.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5118f4d1-704f-4d21-a89e-104f3f7a8a4d -i td/test-erasure-code-plugins/0/new.json 2026-02-22T21:49:19.520 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:49:19.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2026-02-22T21:49:19.529 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQDfeZtpP2/OFxAAG0jc8D8An7kw+/q06Wsv/w== --osd-uuid 5118f4d1-704f-4d21-a89e-104f3f7a8a4d 2026-02-22T21:49:19.544 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:19.531+0000 7f3a30ddf8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:19.545 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:19.531+0000 7f3a30ddf8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:19.546 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:19.535+0000 7f3a30ddf8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:19.547 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:19.535+0000 7f3a30ddf8c0 -1 bdev(0x56411986dc00 td/test-erasure-code-plugins/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:49:19.547 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:19.535+0000 7f3a30ddf8c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2026-02-22T21:49:22.383 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2026-02-22T21:49:22.383 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:49:22.383 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:49:22.384 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:49:22.384 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:49:22.647 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:49:22.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:49:22.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8da4ca76-72c8-47cb-ac83-785f96de51ad --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.12473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2026-02-22T21:49:22.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:49:22.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:49:22.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:49:22.668 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:22.655+0000 7f891e1c78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:22.670 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:22.659+0000 7f891e1c78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:22.671 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:22.659+0000 7f891e1c78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:22.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:49:22.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:49:22.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:49:22.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:49:22.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:49:22.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:22.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:49:22.851 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:49:22.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:22.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:23.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:24.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:24.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:24.047 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:49:24.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:49:24.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:24.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:24.167 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:24.155+0000 7f891e1c78c0 -1 Falling back to public interface 2026-02-22T21:49:24.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:25.239 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:25.239 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:25.239 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:49:25.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:49:25.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:25.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:25.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:26.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:26.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:26.435 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:49:26.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:49:26.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:26.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:26.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:27.225 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:27.211+0000 7f891e1c78c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:49:27.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:27.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:27.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:49:27.634 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:49:27.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:27.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:27.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:28.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:28.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:28.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:49:28.828 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:49:28.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:28.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:29.017 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3404551963,v1:127.0.0.1:6803/3404551963] [v2:127.0.0.1:6804/3404551963,v1:127.0.0.1:6805/3404551963] exists,up 5118f4d1-704f-4d21-a89e-104f3f7a8a4d 2026-02-22T21:49:29.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:49:29.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:49:29.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:49:29.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2026-02-22T21:49:29.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:49:29.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:49:29.019 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:29.019 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:29.019 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:29.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.12473/ceph-osd.0.asok 2026-02-22T21:49:29.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2026-02-22T21:49:29.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.12473/ceph-osd.0.asok log flush 2026-02-22T21:49:29.078 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:49:29.083 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2026-02-22T21:49:29.085 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:49:19.079+0000 7fcb0d510ec0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-22T21:49:29.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2026-02-22T21:49:29.087 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:49:24.155+0000 7f891e1c78c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2026-02-22T21:49:29.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2026-02-22T21:49:29.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:29.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:49:29.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:49:29.089 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:49:29.090 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:49:29.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:49:29.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:49:29.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:49:29.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:49:29.197 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:49:29.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:49:29.198 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:49:29.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:49:29.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:49:29.199 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:49:29.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:29.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:49:29.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:49:29.201 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:29.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:49:29.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:49:29.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:49:29.206 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:49:29.206 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:29.206 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:29.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:49:29.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:49:29.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:49:29.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2026-02-22T21:49:29.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2377: main: code=0 2026-02-22T21:49:29.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: main: teardown td/test-erasure-code-plugins 0 2026-02-22T21:49:29.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2026-02-22T21:49:29.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2026-02-22T21:49:29.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2026-02-22T21:49:29.209 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:49:29.209 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:49:29.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:49:29.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:49:29.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:49:29.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:49:29.211 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:49:29.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:49:29.212 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:49:29.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:49:29.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:49:29.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:49:29.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:29.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:49:29.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:49:29.215 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:29.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:49:29.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T21:49:29.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2026-02-22T21:49:29.218 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:49:29.218 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:29.218 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.12473 2026-02-22T21:49:29.218 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.12473 2026-02-22T21:49:29.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:49:29.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:49:29.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2382: main: return 0 2026-02-22T21:49:29.220 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T21:49:29.220 DEBUG:teuthology.orchestra.run.trial034:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T21:49:29.229 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2026-02-22T21:49:29.229 DEBUG:teuthology.orchestra.run.trial034:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2026-02-22T21:49:29.279 INFO:tasks.workunit.client.0.trial034.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T21:49:29.282 INFO:tasks.workunit.client.0.trial034.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T21:49:29.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T21:49:29.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T21:49:29.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: export PYTHONWARNINGS=ignore 2026-02-22T21:49:29.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: PYTHONWARNINGS=ignore 2026-02-22T21:49:29.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: export CEPH_CONF=/dev/null 2026-02-22T21:49:29.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: CEPH_CONF=/dev/null 2026-02-22T21:49:29.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: main: unset CEPH_ARGS 2026-02-22T21:49:29.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: main: local code 2026-02-22T21:49:29.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: main: run td/test-erasure-code 2026-02-22T21:49:29.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2026-02-22T21:49:29.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2026-02-22T21:49:29.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2026-02-22T21:49:29.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2026-02-22T21:49:29.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2026-02-22T21:49:29.283 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2026-02-22T21:49:29.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none ' 2026-02-22T21:49:29.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:49:29.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2026-02-22T21:49:29.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code 2026-02-22T21:49:29.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code 2026-02-22T21:49:29.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2026-02-22T21:49:29.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:49:29.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2026-02-22T21:49:29.286 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:49:29.286 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:49:29.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:49:29.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:49:29.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:49:29.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:49:29.287 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:49:29.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:49:29.287 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:49:29.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:49:29.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:49:29.289 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:49:29.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:29.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:49:29.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:49:29.290 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:49:29.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:49:29.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:49:29.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2026-02-22T21:49:29.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:49:29.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:29.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:49:29.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.42545 2026-02-22T21:49:29.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:49:29.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:49:29.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code 2026-02-22T21:49:29.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:49:29.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:29.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:49:29.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.42545 2026-02-22T21:49:29.299 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:49:29.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2026-02-22T21:49:29.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2026-02-22T21:49:29.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:49:29.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2026-02-22T21:49:29.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2026-02-22T21:49:29.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code 2026-02-22T21:49:29.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:49:29.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:49:29.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:49:29.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code/a 2026-02-22T21:49:29.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2026-02-22T21:49:29.322 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:49:29.323 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:29.323 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:29.323 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:29.323 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:29.323 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:49:29.323 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:49:29.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T21:49:29.352 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:49:29.354 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:49:29.354 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:49:29.355 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:49:29.355 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:49:29.355 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:49:29.355 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:49:29.355 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:49:29.355 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:49:29.355 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:29.355 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:29.355 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:49:29.355 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.42545/ceph-mon.a.asok 2026-02-22T21:49:29.355 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:49:29.356 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.42545/ceph-mon.a.asok config get fsid 2026-02-22T21:49:29.409 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:49:29.409 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:49:29.410 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:49:29.410 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:49:29.410 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:49:29.410 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:49:29.410 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:49:29.410 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:49:29.411 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:29.411 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:29.411 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:49:29.411 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.42545/ceph-mon.a.asok 2026-02-22T21:49:29.411 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:49:29.412 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.42545/ceph-mon.a.asok config get mon_host 2026-02-22T21:49:29.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2026-02-22T21:49:29.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code 2026-02-22T21:49:29.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:49:29.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:49:29.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:49:29.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code/x 2026-02-22T21:49:29.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:49:29.573 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:49:29.573 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:29.573 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:29.574 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:29.574 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:29.574 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:49:29.574 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:49:29.574 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:49:29.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:49:29.593 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2026-02-22T21:49:29.594 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:49:29.594 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:49:29.594 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:49:29.594 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:29.594 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:49:29.594 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.42545/ceph-mon.a.asok 2026-02-22T21:49:29.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2026-02-22T21:49:29.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.42545/ceph-mon.a.asok log flush 2026-02-22T21:49:29.643 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:49:29.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2026-02-22T21:49:29.650 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:49:29.331+0000 7f956a9dcec0 0 load: jerasure load: lrc 2026-02-22T21:49:29.651 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2026-02-22T21:49:29.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-22T21:49:29.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2026-02-22T21:49:29.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2026-02-22T21:49:29.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:49:29.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:49:29.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:49:29.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/0 2026-02-22T21:49:29.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:49:29.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:49:29.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:49:29.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:49:29.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2026-02-22T21:49:29.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2026-02-22T21:49:29.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:49:29.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:49:29.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:49:29.656 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:49:29.656 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:29.656 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:29.656 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:29.656 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:29.657 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:49:29.657 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:49:29.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:49:29.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:49:29.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:49:29.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:49:29.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:49:29.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:49:29.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:49:29.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:49:29.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:49:29.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:49:29.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:49:29.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:49:29.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/0 2026-02-22T21:49:29.659 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:49:29.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=10f19b16-097e-4b23-b90d-8a42989c8112 2026-02-22T21:49:29.660 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 10f19b16-097e-4b23-b90d-8a42989c8112 2026-02-22T21:49:29.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 10f19b16-097e-4b23-b90d-8a42989c8112' 2026-02-22T21:49:29.661 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:49:29.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpeZtptEQFJxAAh/xm+OQqZhPPdSXiCf/JAg== 2026-02-22T21:49:29.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpeZtptEQFJxAAh/xm+OQqZhPPdSXiCf/JAg=="}' 2026-02-22T21:49:29.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 10f19b16-097e-4b23-b90d-8a42989c8112 -i td/test-erasure-code/0/new.json 2026-02-22T21:49:29.772 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:49:29.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/0/new.json 2026-02-22T21:49:29.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDpeZtptEQFJxAAh/xm+OQqZhPPdSXiCf/JAg== --osd-uuid 10f19b16-097e-4b23-b90d-8a42989c8112 2026-02-22T21:49:29.795 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:29.783+0000 7f3d1bd9b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:29.797 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:29.783+0000 7f3d1bd9b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:29.798 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:29.787+0000 7f3d1bd9b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:29.798 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:29.787+0000 7f3d1bd9b8c0 -1 bdev(0x5574167bbc00 td/test-erasure-code/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:49:29.798 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:29.787+0000 7f3d1bd9b8c0 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2026-02-22T21:49:32.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/0/keyring 2026-02-22T21:49:32.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:49:32.607 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:49:32.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:49:32.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:49:32.877 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:49:32.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:49:32.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:49:32.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:49:32.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:49:32.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:49:32.900 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:32.887+0000 7f588bdd88c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:32.902 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:32.891+0000 7f588bdd88c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:32.903 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:32.891+0000 7f588bdd88c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:33.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:49:33.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:49:33.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:49:33.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:49:33.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:49:33.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:33.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:49:33.069 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:49:33.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:33.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:33.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:34.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:34.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:34.267 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:49:34.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:49:34.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:34.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:34.411 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:34.399+0000 7f588bdd88c0 -1 Falling back to public interface 2026-02-22T21:49:34.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:35.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:35.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:35.469 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:49:35.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:49:35.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:35.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:35.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:36.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:36.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:36.674 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:49:36.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:49:36.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:36.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:36.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:37.436 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:37.423+0000 7f588bdd88c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:49:37.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:37.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:37.873 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:49:37.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:49:37.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:37.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:38.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:39.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:39.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:39.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:49:39.066 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:49:39.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:39.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:49:39.271 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4086348494,v1:127.0.0.1:6803/4086348494] [v2:127.0.0.1:6804/4086348494,v1:127.0.0.1:6805/4086348494] exists,up 10f19b16-097e-4b23-b90d-8a42989c8112 2026-02-22T21:49:39.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:49:39.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:49:39.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:49:39.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-22T21:49:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2026-02-22T21:49:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2026-02-22T21:49:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:49:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T21:49:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:49:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/1 2026-02-22T21:49:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:49:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:49:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:49:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:49:39.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2026-02-22T21:49:39.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2026-02-22T21:49:39.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:49:39.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:49:39.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:49:39.277 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:49:39.277 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:39.277 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:39.277 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:39.277 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:39.277 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:49:39.277 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:49:39.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/1 2026-02-22T21:49:39.279 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:49:39.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=425eb34b-33d6-4044-b13d-4657bfaa62d1 2026-02-22T21:49:39.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 425eb34b-33d6-4044-b13d-4657bfaa62d1' 2026-02-22T21:49:39.279 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 425eb34b-33d6-4044-b13d-4657bfaa62d1 2026-02-22T21:49:39.279 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:49:39.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzeZtpkR1vEBAAAbxhPC6DoCUTROPTXC9pNg== 2026-02-22T21:49:39.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzeZtpkR1vEBAAAbxhPC6DoCUTROPTXC9pNg=="}' 2026-02-22T21:49:39.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 425eb34b-33d6-4044-b13d-4657bfaa62d1 -i td/test-erasure-code/1/new.json 2026-02-22T21:49:39.495 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:49:39.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/1/new.json 2026-02-22T21:49:39.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzeZtpkR1vEBAAAbxhPC6DoCUTROPTXC9pNg== --osd-uuid 425eb34b-33d6-4044-b13d-4657bfaa62d1 2026-02-22T21:49:39.521 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:39.507+0000 7eff9d1aa8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:39.523 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:39.511+0000 7eff9d1aa8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:39.524 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:39.511+0000 7eff9d1aa8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:39.524 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:39.511+0000 7eff9d1aa8c0 -1 bdev(0x5639782abc00 td/test-erasure-code/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:49:39.525 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:39.511+0000 7eff9d1aa8c0 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2026-02-22T21:49:41.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/1/keyring 2026-02-22T21:49:41.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:49:41.800 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T21:49:41.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:49:41.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:49:42.068 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T21:49:42.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T21:49:42.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:49:42.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:49:42.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:49:42.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:49:42.083 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:42.071+0000 7fe93014b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:42.085 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:42.071+0000 7fe93014b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:42.086 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:42.075+0000 7fe93014b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:42.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T21:49:42.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:49:42.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T21:49:42.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:49:42.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:49:42.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:42.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:49:42.271 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:49:42.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:42.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:49:42.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:43.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:43.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:43.471 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:49:43.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:49:43.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:43.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:49:43.587 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:43.575+0000 7fe93014b8c0 -1 Falling back to public interface 2026-02-22T21:49:43.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:44.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:44.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:44.665 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:49:44.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:49:44.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:44.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:49:44.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:45.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:45.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:45.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:49:45.870 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:49:45.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:45.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:49:46.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:46.638 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:46.627+0000 7fe93014b8c0 -1 osd.1 0 log_to_monitors true 2026-02-22T21:49:47.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:47.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:47.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:49:47.068 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:49:47.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:47.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:49:47.268 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/864350738,v1:127.0.0.1:6811/864350738] [v2:127.0.0.1:6812/864350738,v1:127.0.0.1:6813/864350738] exists,up 425eb34b-33d6-4044-b13d-4657bfaa62d1 2026-02-22T21:49:47.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:49:47.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:49:47.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:49:47.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-22T21:49:47.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2026-02-22T21:49:47.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2026-02-22T21:49:47.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:49:47.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T21:49:47.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:49:47.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/2 2026-02-22T21:49:47.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:49:47.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:49:47.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:49:47.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:49:47.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2026-02-22T21:49:47.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2026-02-22T21:49:47.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:49:47.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:49:47.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:49:47.273 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:49:47.273 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:47.273 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:47.273 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:47.273 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:47.273 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:49:47.273 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:49:47.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:49:47.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:49:47.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:49:47.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:49:47.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:49:47.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:49:47.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:49:47.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:49:47.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:49:47.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:49:47.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:49:47.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:49:47.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/2 2026-02-22T21:49:47.275 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:49:47.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=36120b78-ddbb-4dab-b331-9d81c43c49db 2026-02-22T21:49:47.275 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 36120b78-ddbb-4dab-b331-9d81c43c49db 2026-02-22T21:49:47.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 36120b78-ddbb-4dab-b331-9d81c43c49db' 2026-02-22T21:49:47.276 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:49:47.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7eZtpJY9cEBAAGFb6027QXtOflvoBmWpZ+w== 2026-02-22T21:49:47.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7eZtpJY9cEBAAGFb6027QXtOflvoBmWpZ+w=="}' 2026-02-22T21:49:47.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 36120b78-ddbb-4dab-b331-9d81c43c49db -i td/test-erasure-code/2/new.json 2026-02-22T21:49:47.493 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:49:47.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/2/new.json 2026-02-22T21:49:47.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7eZtpJY9cEBAAGFb6027QXtOflvoBmWpZ+w== --osd-uuid 36120b78-ddbb-4dab-b331-9d81c43c49db 2026-02-22T21:49:47.519 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:47.507+0000 7ff2247fd8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:47.520 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:47.507+0000 7ff2247fd8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:47.521 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:47.511+0000 7ff2247fd8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:47.522 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:47.511+0000 7ff2247fd8c0 -1 bdev(0x557b59253c00 td/test-erasure-code/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:49:47.522 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:47.511+0000 7ff2247fd8c0 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2026-02-22T21:49:49.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/2/keyring 2026-02-22T21:49:49.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:49:49.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:49:49.782 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T21:49:49.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:49:50.050 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T21:49:50.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T21:49:50.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:49:50.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:49:50.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:49:50.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:49:50.074 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:50.064+0000 7f234675e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:50.075 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:50.064+0000 7f234675e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:50.077 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:50.068+0000 7f234675e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:50.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T21:49:50.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:49:50.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T21:49:50.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:49:50.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:49:50.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:50.253 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:49:50.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:49:50.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:50.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:49:50.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:51.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:51.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:51.458 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:49:51.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:49:51.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:51.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:49:51.566 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:51.556+0000 7f234675e8c0 -1 Falling back to public interface 2026-02-22T21:49:51.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:52.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:52.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:52.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:49:52.656 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:49:52.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:52.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:49:52.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:53.857 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:53.857 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:53.857 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:49:53.858 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:49:53.858 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:53.858 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:49:54.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:54.579 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:54.564+0000 7f234675e8c0 -1 osd.2 0 log_to_monitors true 2026-02-22T21:49:55.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:55.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:55.054 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:49:55.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:49:55.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:55.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:49:55.243 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4288840868,v1:127.0.0.1:6819/4288840868] [v2:127.0.0.1:6820/4288840868,v1:127.0.0.1:6821/4288840868] exists,up 36120b78-ddbb-4dab-b331-9d81c43c49db 2026-02-22T21:49:55.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:49:55.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:49:55.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:49:55.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-22T21:49:55.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2026-02-22T21:49:55.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2026-02-22T21:49:55.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:49:55.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T21:49:55.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:49:55.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/3 2026-02-22T21:49:55.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:49:55.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:49:55.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:49:55.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:49:55.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2026-02-22T21:49:55.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2026-02-22T21:49:55.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:49:55.248 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:49:55.248 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:49:55.248 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:49:55.248 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:49:55.248 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:49:55.248 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:49:55.248 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:49:55.248 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:49:55.248 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:49:55.248 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:49:55.248 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:49:55.248 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:49:55.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:49:55.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:49:55.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:49:55.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:49:55.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:49:55.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:49:55.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:49:55.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:49:55.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:49:55.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/3 2026-02-22T21:49:55.250 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:49:55.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c6cf897-2094-44c0-8256-2f77e5a5fcf4 2026-02-22T21:49:55.250 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 5c6cf897-2094-44c0-8256-2f77e5a5fcf4 2026-02-22T21:49:55.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5c6cf897-2094-44c0-8256-2f77e5a5fcf4' 2026-02-22T21:49:55.251 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:49:55.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADeptpiEUIDxAAJyMhm1rXL+BkE9M4mNj6wA== 2026-02-22T21:49:55.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADeptpiEUIDxAAJyMhm1rXL+BkE9M4mNj6wA=="}' 2026-02-22T21:49:55.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c6cf897-2094-44c0-8256-2f77e5a5fcf4 -i td/test-erasure-code/3/new.json 2026-02-22T21:49:55.463 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:49:55.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/3/new.json 2026-02-22T21:49:55.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADeptpiEUIDxAAJyMhm1rXL+BkE9M4mNj6wA== --osd-uuid 5c6cf897-2094-44c0-8256-2f77e5a5fcf4 2026-02-22T21:49:55.488 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:55.477+0000 7f4309f228c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:55.490 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:55.477+0000 7f4309f228c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:55.491 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:55.477+0000 7f4309f228c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:55.491 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:55.481+0000 7f4309f228c0 -1 bdev(0x5578024edc00 td/test-erasure-code/3/block) open stat got: (1) Operation not permitted 2026-02-22T21:49:55.491 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:55.481+0000 7f4309f228c0 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2026-02-22T21:49:57.763 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/3/keyring 2026-02-22T21:49:57.763 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:49:57.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T21:49:57.765 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T21:49:57.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:49:58.031 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T21:49:58.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T21:49:58.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:49:58.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:49:58.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:49:58.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:49:58.047 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:58.037+0000 7f5e31bfa8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:58.049 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:58.037+0000 7f5e31bfa8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:58.050 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:58.037+0000 7f5e31bfa8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:49:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T21:49:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:49:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T21:49:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:49:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:49:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:58.226 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:49:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:49:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:58.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:49:58.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:49:59.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:49:59.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:49:59.430 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:49:59.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:49:59.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:49:59.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:49:59.524 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:49:59.513+0000 7f5e31bfa8c0 -1 Falling back to public interface 2026-02-22T21:49:59.630 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:00.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:00.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:00.631 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:50:00.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:50:00.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:00.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:50:00.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:01.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:01.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:01.830 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:50:01.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:50:01.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:01.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:50:02.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:02.830 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:02.822+0000 7f5e31bfa8c0 -1 osd.3 0 log_to_monitors true 2026-02-22T21:50:03.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:03.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:03.031 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:50:03.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:50:03.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:03.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:50:03.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:04.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:04.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:04.234 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:50:04.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:50:04.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:04.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:50:04.436 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1714631299,v1:127.0.0.1:6827/1714631299] [v2:127.0.0.1:6828/1714631299,v1:127.0.0.1:6829/1714631299] exists,up 5c6cf897-2094-44c0-8256-2f77e5a5fcf4 2026-02-22T21:50:04.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:50:04.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:50:04.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:50:04.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-22T21:50:04.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2026-02-22T21:50:04.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2026-02-22T21:50:04.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:50:04.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2026-02-22T21:50:04.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:50:04.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/4 2026-02-22T21:50:04.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:50:04.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:50:04.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:50:04.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:50:04.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2026-02-22T21:50:04.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2026-02-22T21:50:04.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:50:04.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:50:04.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:50:04.442 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:50:04.442 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:50:04.442 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:50:04.442 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:50:04.442 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:50:04.442 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:50:04.442 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:04.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:04.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:50:04.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:50:04.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:50:04.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:50:04.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:50:04.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:50:04.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:50:04.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:50:04.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:50:04.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:50:04.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:50:04.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/4 2026-02-22T21:50:04.446 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:50:04.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b3dae501-9f80-4100-b263-e212c51f196c 2026-02-22T21:50:04.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 b3dae501-9f80-4100-b263-e212c51f196c' 2026-02-22T21:50:04.447 INFO:tasks.workunit.client.0.trial034.stdout:add osd4 b3dae501-9f80-4100-b263-e212c51f196c 2026-02-22T21:50:04.447 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:50:04.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAMeptp5delGhAAwfma7uF4s23F5KK/gxAsIA== 2026-02-22T21:50:04.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAMeptp5delGhAAwfma7uF4s23F5KK/gxAsIA=="}' 2026-02-22T21:50:04.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b3dae501-9f80-4100-b263-e212c51f196c -i td/test-erasure-code/4/new.json 2026-02-22T21:50:04.660 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:50:04.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/4/new.json 2026-02-22T21:50:04.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMeptp5delGhAAwfma7uF4s23F5KK/gxAsIA== --osd-uuid b3dae501-9f80-4100-b263-e212c51f196c 2026-02-22T21:50:04.686 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:04.678+0000 7f7c6addb8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:04.687 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:04.678+0000 7f7c6addb8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:04.688 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:04.678+0000 7f7c6addb8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:04.689 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:04.678+0000 7f7c6addb8c0 -1 bdev(0x55fe955b3c00 td/test-erasure-code/4/block) open stat got: (1) Operation not permitted 2026-02-22T21:50:04.689 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:04.678+0000 7f7c6addb8c0 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2026-02-22T21:50:06.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/4/keyring 2026-02-22T21:50:06.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:50:06.951 INFO:tasks.workunit.client.0.trial034.stdout:adding osd4 key to auth repository 2026-02-22T21:50:06.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2026-02-22T21:50:06.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:50:07.220 INFO:tasks.workunit.client.0.trial034.stdout:start osd.4 2026-02-22T21:50:07.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2026-02-22T21:50:07.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:50:07.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:50:07.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:50:07.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:50:07.242 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:07.234+0000 7f00b85498c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:07.244 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:07.234+0000 7f00b85498c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:07.245 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:07.238+0000 7f00b85498c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:07.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2026-02-22T21:50:07.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:50:07.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2026-02-22T21:50:07.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:50:07.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:50:07.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:07.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:50:07.421 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:50:07.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:07.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:50:07.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:08.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:08.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:08.622 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:50:08.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:50:08.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:08.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:50:08.799 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:08.791+0000 7f00b85498c0 -1 Falling back to public interface 2026-02-22T21:50:08.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:09.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:09.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:09.823 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:50:09.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:50:09.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:09.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:50:10.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:11.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:11.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:11.020 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:50:11.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:50:11.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:11.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:50:11.218 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:12.047 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:12.039+0000 7f00b85498c0 -1 osd.4 0 log_to_monitors true 2026-02-22T21:50:12.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:12.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:12.220 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:50:12.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:50:12.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:12.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:50:12.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:13.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:13.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:13.422 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:50:13.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:50:13.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:13.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:50:13.618 INFO:tasks.workunit.client.0.trial034.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2791560490,v1:127.0.0.1:6835/2791560490] [v2:127.0.0.1:6836/2791560490,v1:127.0.0.1:6837/2791560490] exists,up b3dae501-9f80-4100-b263-e212c51f196c 2026-02-22T21:50:13.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:50:13.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:50:13.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:50:13.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-22T21:50:13.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2026-02-22T21:50:13.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2026-02-22T21:50:13.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:50:13.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2026-02-22T21:50:13.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:50:13.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/5 2026-02-22T21:50:13.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:50:13.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:50:13.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:50:13.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:50:13.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2026-02-22T21:50:13.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2026-02-22T21:50:13.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:50:13.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:50:13.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:50:13.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:50:13.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:50:13.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:50:13.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:50:13.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:50:13.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:50:13.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:50:13.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/5 2026-02-22T21:50:13.625 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:50:13.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c21beadb-f200-42f4-8059-76a7ad8f17d4 2026-02-22T21:50:13.625 INFO:tasks.workunit.client.0.trial034.stdout:add osd5 c21beadb-f200-42f4-8059-76a7ad8f17d4 2026-02-22T21:50:13.626 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 c21beadb-f200-42f4-8059-76a7ad8f17d4' 2026-02-22T21:50:13.626 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:50:13.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVeptp9EuJJRAAS0aluE/HlJDathiOrM+Jyg== 2026-02-22T21:50:13.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVeptp9EuJJRAAS0aluE/HlJDathiOrM+Jyg=="}' 2026-02-22T21:50:13.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c21beadb-f200-42f4-8059-76a7ad8f17d4 -i td/test-erasure-code/5/new.json 2026-02-22T21:50:13.842 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:50:13.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/5/new.json 2026-02-22T21:50:13.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVeptp9EuJJRAAS0aluE/HlJDathiOrM+Jyg== --osd-uuid c21beadb-f200-42f4-8059-76a7ad8f17d4 2026-02-22T21:50:13.867 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:13.859+0000 7f69c39408c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:13.869 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:13.863+0000 7f69c39408c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:13.870 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:13.863+0000 7f69c39408c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:13.870 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:13.863+0000 7f69c39408c0 -1 bdev(0x5611f054dc00 td/test-erasure-code/5/block) open stat got: (1) Operation not permitted 2026-02-22T21:50:13.871 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:13.863+0000 7f69c39408c0 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2026-02-22T21:50:16.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/5/keyring 2026-02-22T21:50:16.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:50:16.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2026-02-22T21:50:16.149 INFO:tasks.workunit.client.0.trial034.stdout:adding osd5 key to auth repository 2026-02-22T21:50:16.150 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:50:16.422 INFO:tasks.workunit.client.0.trial034.stdout:start osd.5 2026-02-22T21:50:16.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2026-02-22T21:50:16.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:50:16.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:50:16.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:50:16.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:50:16.445 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:16.439+0000 7fa235db98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:16.446 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:16.439+0000 7fa235db98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:16.447 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:16.439+0000 7fa235db98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:16.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2026-02-22T21:50:16.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:50:16.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2026-02-22T21:50:16.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:50:16.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:50:16.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:16.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:50:16.619 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:50:16.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:16.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:50:16.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:17.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:17.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:17.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:50:17.820 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:50:17.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:17.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:50:17.910 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:17.903+0000 7fa235db98c0 -1 Falling back to public interface 2026-02-22T21:50:18.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:19.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:19.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:19.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:50:19.017 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:50:19.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:19.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:50:19.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:20.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:20.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:20.213 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:50:20.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:50:20.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:20.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:50:20.415 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:21.111 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:21.104+0000 7fa235db98c0 -1 osd.5 0 log_to_monitors true 2026-02-22T21:50:21.416 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:21.416 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:21.416 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:50:21.416 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:50:21.416 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:21.416 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:50:21.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:22.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:22.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:22.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:50:22.622 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:50:22.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:22.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:50:22.819 INFO:tasks.workunit.client.0.trial034.stdout:osd.5 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/141264883,v1:127.0.0.1:6843/141264883] [v2:127.0.0.1:6844/141264883,v1:127.0.0.1:6845/141264883] exists,up c21beadb-f200-42f4-8059-76a7ad8f17d4 2026-02-22T21:50:22.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:50:22.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:50:22.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:50:22.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-22T21:50:22.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2026-02-22T21:50:22.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2026-02-22T21:50:22.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:50:22.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2026-02-22T21:50:22.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:50:22.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/6 2026-02-22T21:50:22.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:50:22.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:50:22.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:50:22.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:50:22.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2026-02-22T21:50:22.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2026-02-22T21:50:22.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:50:22.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:50:22.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:50:22.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:50:22.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:50:22.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:50:22.828 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:50:22.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:50:22.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:50:22.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:50:22.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/6 2026-02-22T21:50:22.830 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:50:22.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e712364e-e3c5-4f70-8962-81b340e84b13 2026-02-22T21:50:22.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 e712364e-e3c5-4f70-8962-81b340e84b13' 2026-02-22T21:50:22.830 INFO:tasks.workunit.client.0.trial034.stdout:add osd6 e712364e-e3c5-4f70-8962-81b340e84b13 2026-02-22T21:50:22.831 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:50:22.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAeeptpIiiGMRAAMMnv/dXvT7JVOG6DjTwUzA== 2026-02-22T21:50:22.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAeeptpIiiGMRAAMMnv/dXvT7JVOG6DjTwUzA=="}' 2026-02-22T21:50:22.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e712364e-e3c5-4f70-8962-81b340e84b13 -i td/test-erasure-code/6/new.json 2026-02-22T21:50:23.031 INFO:tasks.workunit.client.0.trial034.stdout:6 2026-02-22T21:50:23.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/6/new.json 2026-02-22T21:50:23.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAeeptpIiiGMRAAMMnv/dXvT7JVOG6DjTwUzA== --osd-uuid e712364e-e3c5-4f70-8962-81b340e84b13 2026-02-22T21:50:23.056 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:23.052+0000 7effddb3d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:23.057 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:23.052+0000 7effddb3d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:23.059 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:23.052+0000 7effddb3d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:23.059 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:23.052+0000 7effddb3d8c0 -1 bdev(0x55dd1c1dbc00 td/test-erasure-code/6/block) open stat got: (1) Operation not permitted 2026-02-22T21:50:23.060 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:23.052+0000 7effddb3d8c0 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2026-02-22T21:50:25.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/6/keyring 2026-02-22T21:50:25.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:50:25.337 INFO:tasks.workunit.client.0.trial034.stdout:adding osd6 key to auth repository 2026-02-22T21:50:25.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2026-02-22T21:50:25.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:50:25.609 INFO:tasks.workunit.client.0.trial034.stdout:start osd.6 2026-02-22T21:50:25.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2026-02-22T21:50:25.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:50:25.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:50:25.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:50:25.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:50:25.630 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:25.620+0000 7fd2b7d318c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:25.632 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:25.624+0000 7fd2b7d318c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:25.633 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:25.624+0000 7fd2b7d318c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:25.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2026-02-22T21:50:25.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:50:25.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2026-02-22T21:50:25.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:50:25.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:50:25.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:25.805 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:50:25.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:50:25.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:25.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:50:26.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:27.005 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:27.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:27.006 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:50:27.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:50:27.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:27.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:50:27.133 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:27.124+0000 7fd2b7d318c0 -1 Falling back to public interface 2026-02-22T21:50:27.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:28.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:28.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:28.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:50:28.206 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:50:28.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:28.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:50:28.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:29.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:29.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:29.409 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:50:29.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:50:29.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:29.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:50:29.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:30.230 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:30.220+0000 7fd2b7d318c0 -1 osd.6 0 log_to_monitors true 2026-02-22T21:50:30.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:30.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:30.612 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:50:30.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:50:30.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:30.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:50:30.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:31.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:31.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:31.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:50:31.807 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:50:31.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:31.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:50:32.006 INFO:tasks.workunit.client.0.trial034.stdout:osd.6 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/54708834,v1:127.0.0.1:6851/54708834] [v2:127.0.0.1:6852/54708834,v1:127.0.0.1:6853/54708834] exists,up e712364e-e3c5-4f70-8962-81b340e84b13 2026-02-22T21:50:32.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:50:32.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:50:32.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:50:32.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-22T21:50:32.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2026-02-22T21:50:32.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2026-02-22T21:50:32.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:50:32.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2026-02-22T21:50:32.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:50:32.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/7 2026-02-22T21:50:32.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:50:32.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:50:32.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:50:32.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:50:32.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2026-02-22T21:50:32.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2026-02-22T21:50:32.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:50:32.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:50:32.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:50:32.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:50:32.012 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:50:32.012 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:50:32.012 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:50:32.012 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:50:32.012 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:50:32.012 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:32.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:32.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:50:32.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:50:32.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:50:32.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:50:32.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:50:32.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:50:32.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:50:32.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:50:32.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:50:32.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:50:32.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:50:32.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/7 2026-02-22T21:50:32.014 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:50:32.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8ed09d79-0c53-4733-bec7-4da337e2d940 2026-02-22T21:50:32.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 8ed09d79-0c53-4733-bec7-4da337e2d940' 2026-02-22T21:50:32.015 INFO:tasks.workunit.client.0.trial034.stdout:add osd7 8ed09d79-0c53-4733-bec7-4da337e2d940 2026-02-22T21:50:32.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:50:32.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAoeptpI/QgARAAcwXjBlGhnIGTni+jBzLmMg== 2026-02-22T21:50:32.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAoeptpI/QgARAAcwXjBlGhnIGTni+jBzLmMg=="}' 2026-02-22T21:50:32.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8ed09d79-0c53-4733-bec7-4da337e2d940 -i td/test-erasure-code/7/new.json 2026-02-22T21:50:32.232 INFO:tasks.workunit.client.0.trial034.stdout:7 2026-02-22T21:50:32.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/7/new.json 2026-02-22T21:50:32.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoeptpI/QgARAAcwXjBlGhnIGTni+jBzLmMg== --osd-uuid 8ed09d79-0c53-4733-bec7-4da337e2d940 2026-02-22T21:50:32.256 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:32.248+0000 7f3a8ee368c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:32.258 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:32.248+0000 7f3a8ee368c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:32.259 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:32.252+0000 7f3a8ee368c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:32.259 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:32.252+0000 7f3a8ee368c0 -1 bdev(0x55be6da35c00 td/test-erasure-code/7/block) open stat got: (1) Operation not permitted 2026-02-22T21:50:32.259 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:32.252+0000 7f3a8ee368c0 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2026-02-22T21:50:34.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/7/keyring 2026-02-22T21:50:34.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:50:34.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2026-02-22T21:50:34.528 INFO:tasks.workunit.client.0.trial034.stdout:adding osd7 key to auth repository 2026-02-22T21:50:34.529 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:50:34.805 INFO:tasks.workunit.client.0.trial034.stdout:start osd.7 2026-02-22T21:50:34.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2026-02-22T21:50:34.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:50:34.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:50:34.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:50:34.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:50:34.826 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:34.820+0000 7f2b9bd2a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:34.828 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:34.820+0000 7f2b9bd2a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:34.829 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:34.820+0000 7f2b9bd2a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2026-02-22T21:50:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:50:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2026-02-22T21:50:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:50:34.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:50:34.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:34.998 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:50:34.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:50:34.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:34.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2026-02-22T21:50:35.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:36.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:36.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:36.201 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:50:36.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:50:36.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:36.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2026-02-22T21:50:36.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:36.560 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:36.552+0000 7f2b9bd2a8c0 -1 Falling back to public interface 2026-02-22T21:50:37.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:37.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:37.400 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:50:37.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:50:37.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:37.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2026-02-22T21:50:37.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:38.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:38.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:38.599 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:50:38.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:50:38.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:38.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2026-02-22T21:50:38.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:39.600 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:39.593+0000 7f2b9bd2a8c0 -1 osd.7 0 log_to_monitors true 2026-02-22T21:50:39.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:39.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:39.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:50:39.800 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:50:39.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:39.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2026-02-22T21:50:40.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:41.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:41.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:41.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:50:41.022 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:50:41.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:41.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2026-02-22T21:50:41.219 INFO:tasks.workunit.client.0.trial034.stdout:osd.7 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2223167387,v1:127.0.0.1:6859/2223167387] [v2:127.0.0.1:6860/2223167387,v1:127.0.0.1:6861/2223167387] exists,up 8ed09d79-0c53-4733-bec7-4da337e2d940 2026-02-22T21:50:41.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:50:41.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:50:41.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:50:41.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-22T21:50:41.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2026-02-22T21:50:41.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2026-02-22T21:50:41.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:50:41.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2026-02-22T21:50:41.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:50:41.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/8 2026-02-22T21:50:41.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:50:41.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:50:41.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:50:41.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:50:41.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2026-02-22T21:50:41.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2026-02-22T21:50:41.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:50:41.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:50:41.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:50:41.224 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:50:41.224 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:50:41.224 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:50:41.224 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:50:41.224 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:50:41.224 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:50:41.224 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:41.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:41.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:50:41.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:50:41.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:50:41.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:50:41.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:50:41.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:50:41.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:50:41.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:50:41.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:50:41.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:50:41.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:50:41.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/8 2026-02-22T21:50:41.226 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:50:41.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7d5452dd-785a-49d1-9bd6-d2f9e436af3e 2026-02-22T21:50:41.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 7d5452dd-785a-49d1-9bd6-d2f9e436af3e' 2026-02-22T21:50:41.227 INFO:tasks.workunit.client.0.trial034.stdout:add osd8 7d5452dd-785a-49d1-9bd6-d2f9e436af3e 2026-02-22T21:50:41.227 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:50:41.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxeptp/9niDRAA0QWktkCYUd0Gy2jVztZblg== 2026-02-22T21:50:41.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxeptp/9niDRAA0QWktkCYUd0Gy2jVztZblg=="}' 2026-02-22T21:50:41.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7d5452dd-785a-49d1-9bd6-d2f9e436af3e -i td/test-erasure-code/8/new.json 2026-02-22T21:50:41.435 INFO:tasks.workunit.client.0.trial034.stdout:8 2026-02-22T21:50:41.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/8/new.json 2026-02-22T21:50:41.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxeptp/9niDRAA0QWktkCYUd0Gy2jVztZblg== --osd-uuid 7d5452dd-785a-49d1-9bd6-d2f9e436af3e 2026-02-22T21:50:41.459 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:41.453+0000 7fc7326328c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:41.461 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:41.453+0000 7fc7326328c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:41.462 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:41.453+0000 7fc7326328c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:41.462 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:41.457+0000 7fc7326328c0 -1 bdev(0x55a34889dc00 td/test-erasure-code/8/block) open stat got: (1) Operation not permitted 2026-02-22T21:50:41.462 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:41.457+0000 7fc7326328c0 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2026-02-22T21:50:43.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/8/keyring 2026-02-22T21:50:43.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:50:43.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2026-02-22T21:50:43.746 INFO:tasks.workunit.client.0.trial034.stdout:adding osd8 key to auth repository 2026-02-22T21:50:43.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:50:44.003 INFO:tasks.workunit.client.0.trial034.stdout:start osd.8 2026-02-22T21:50:44.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2026-02-22T21:50:44.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:50:44.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:50:44.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:50:44.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:50:44.025 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:44.017+0000 7fc159faf8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:44.027 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:44.021+0000 7fc159faf8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:44.028 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:44.021+0000 7fc159faf8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:44.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2026-02-22T21:50:44.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:50:44.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2026-02-22T21:50:44.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:50:44.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:50:44.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:44.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:50:44.206 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:50:44.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:44.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2026-02-22T21:50:44.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:45.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:45.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:45.404 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:50:45.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:50:45.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:45.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2026-02-22T21:50:45.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:45.800 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:45.793+0000 7fc159faf8c0 -1 Falling back to public interface 2026-02-22T21:50:46.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:46.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:46.597 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:50:46.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:50:46.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:46.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2026-02-22T21:50:46.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:47.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:47.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:47.788 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:50:47.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:50:47.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:47.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2026-02-22T21:50:47.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:48.823 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:48.817+0000 7fc159faf8c0 -1 osd.8 0 log_to_monitors true 2026-02-22T21:50:48.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:48.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:48.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:50:48.980 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:50:48.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:48.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2026-02-22T21:50:49.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:50.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:50.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:50.177 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:50:50.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:50:50.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:50.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2026-02-22T21:50:50.374 INFO:tasks.workunit.client.0.trial034.stdout:osd.8 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1452349694,v1:127.0.0.1:6867/1452349694] [v2:127.0.0.1:6868/1452349694,v1:127.0.0.1:6869/1452349694] exists,up 7d5452dd-785a-49d1-9bd6-d2f9e436af3e 2026-02-22T21:50:50.378 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:50:50.378 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:50:50.378 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:50:50.378 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-22T21:50:50.378 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2026-02-22T21:50:50.378 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2026-02-22T21:50:50.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:50:50.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2026-02-22T21:50:50.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:50:50.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/9 2026-02-22T21:50:50.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:50:50.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:50:50.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:50:50.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:50:50.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2026-02-22T21:50:50.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2026-02-22T21:50:50.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:50:50.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:50:50.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:50:50.380 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:50:50.380 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:50:50.380 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:50:50.380 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:50:50.380 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:50:50.380 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:50:50.380 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:50.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:50.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:50:50.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:50:50.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:50:50.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:50:50.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:50:50.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:50:50.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:50:50.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:50:50.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:50:50.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:50:50.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:50:50.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/9 2026-02-22T21:50:50.381 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:50:50.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4f02bd10-132d-4709-9fc7-eca87e892c93 2026-02-22T21:50:50.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 4f02bd10-132d-4709-9fc7-eca87e892c93' 2026-02-22T21:50:50.383 INFO:tasks.workunit.client.0.trial034.stdout:add osd9 4f02bd10-132d-4709-9fc7-eca87e892c93 2026-02-22T21:50:50.383 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:50:50.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6eptpRVopFxAADwIWxy0+V+rqegzuRqp4Ew== 2026-02-22T21:50:50.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6eptpRVopFxAADwIWxy0+V+rqegzuRqp4Ew=="}' 2026-02-22T21:50:50.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4f02bd10-132d-4709-9fc7-eca87e892c93 -i td/test-erasure-code/9/new.json 2026-02-22T21:50:50.593 INFO:tasks.workunit.client.0.trial034.stdout:9 2026-02-22T21:50:50.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/9/new.json 2026-02-22T21:50:50.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6eptpRVopFxAADwIWxy0+V+rqegzuRqp4Ew== --osd-uuid 4f02bd10-132d-4709-9fc7-eca87e892c93 2026-02-22T21:50:50.627 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:50.621+0000 7f59d0a7c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:50.628 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:50.621+0000 7f59d0a7c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:50.629 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:50.625+0000 7f59d0a7c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:50.630 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:50.625+0000 7f59d0a7c8c0 -1 bdev(0x55591a22dc00 td/test-erasure-code/9/block) open stat got: (1) Operation not permitted 2026-02-22T21:50:50.630 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:50.625+0000 7f59d0a7c8c0 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2026-02-22T21:50:52.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/9/keyring 2026-02-22T21:50:52.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:50:52.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2026-02-22T21:50:52.934 INFO:tasks.workunit.client.0.trial034.stdout:adding osd9 key to auth repository 2026-02-22T21:50:52.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:50:53.191 INFO:tasks.workunit.client.0.trial034.stdout:start osd.9 2026-02-22T21:50:53.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2026-02-22T21:50:53.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:50:53.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:50:53.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:50:53.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:50:53.213 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:53.205+0000 7ff06deca8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:53.215 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:53.209+0000 7ff06deca8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:53.216 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:53.209+0000 7ff06deca8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:53.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2026-02-22T21:50:53.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:50:53.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2026-02-22T21:50:53.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:50:53.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:50:53.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:53.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:50:53.392 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:50:53.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:53.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2026-02-22T21:50:53.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:54.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:54.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:54.588 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:50:54.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:50:54.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:54.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2026-02-22T21:50:54.719 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:54.713+0000 7ff06deca8c0 -1 Falling back to public interface 2026-02-22T21:50:54.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:55.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:55.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:55.782 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:50:55.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:50:55.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:55.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2026-02-22T21:50:55.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:56.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:56.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:56.974 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:50:56.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:50:56.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:56.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2026-02-22T21:50:57.174 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:57.752 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:57.745+0000 7ff06deca8c0 -1 osd.9 0 log_to_monitors true 2026-02-22T21:50:58.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:58.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:58.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:50:58.176 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:50:58.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:58.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2026-02-22T21:50:58.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:50:59.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:50:59.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:50:59.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:50:59.388 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:50:59.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:50:59.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2026-02-22T21:50:59.581 INFO:tasks.workunit.client.0.trial034.stdout:osd.9 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2276935659,v1:127.0.0.1:6875/2276935659] [v2:127.0.0.1:6876/2276935659,v1:127.0.0.1:6877/2276935659] exists,up 4f02bd10-132d-4709-9fc7-eca87e892c93 2026-02-22T21:50:59.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:50:59.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:50:59.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:50:59.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2026-02-22T21:50:59.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2026-02-22T21:50:59.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2026-02-22T21:50:59.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:50:59.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=10 2026-02-22T21:50:59.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:50:59.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/10 2026-02-22T21:50:59.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:50:59.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:50:59.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:50:59.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:50:59.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2026-02-22T21:50:59.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2026-02-22T21:50:59.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:50:59.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:50:59.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:50:59.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:50:59.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:50:59.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:50:59.584 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:50:59.584 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:50:59.585 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:50:59.586 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:59.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:50:59.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:50:59.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:50:59.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:50:59.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:50:59.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:50:59.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:50:59.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:50:59.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:50:59.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:50:59.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:50:59.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:50:59.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/10 2026-02-22T21:50:59.588 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:50:59.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=186013bd-1dfa-4942-b1b1-68f394ee6968 2026-02-22T21:50:59.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd10 186013bd-1dfa-4942-b1b1-68f394ee6968' 2026-02-22T21:50:59.588 INFO:tasks.workunit.client.0.trial034.stdout:add osd10 186013bd-1dfa-4942-b1b1-68f394ee6968 2026-02-22T21:50:59.588 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:50:59.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDeptpmleTIxAANwtjb8+mMuodiHCBdhe+wQ== 2026-02-22T21:50:59.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDeptpmleTIxAANwtjb8+mMuodiHCBdhe+wQ=="}' 2026-02-22T21:50:59.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 186013bd-1dfa-4942-b1b1-68f394ee6968 -i td/test-erasure-code/10/new.json 2026-02-22T21:50:59.802 INFO:tasks.workunit.client.0.trial034.stdout:10 2026-02-22T21:50:59.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/10/new.json 2026-02-22T21:50:59.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 10 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDeptpmleTIxAANwtjb8+mMuodiHCBdhe+wQ== --osd-uuid 186013bd-1dfa-4942-b1b1-68f394ee6968 2026-02-22T21:50:59.828 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:59.821+0000 7f6c2fe4b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:59.830 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:59.825+0000 7f6c2fe4b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:59.831 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:59.825+0000 7f6c2fe4b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:50:59.831 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:59.825+0000 7f6c2fe4b8c0 -1 bdev(0x5584066dfc00 td/test-erasure-code/10/block) open stat got: (1) Operation not permitted 2026-02-22T21:50:59.831 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:50:59.825+0000 7f6c2fe4b8c0 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2026-02-22T21:51:02.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/10/keyring 2026-02-22T21:51:02.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:51:02.103 INFO:tasks.workunit.client.0.trial034.stdout:adding osd10 key to auth repository 2026-02-22T21:51:02.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd10 key to auth repository 2026-02-22T21:51:02.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:51:02.375 INFO:tasks.workunit.client.0.trial034.stdout:start osd.10 2026-02-22T21:51:02.375 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.10 2026-02-22T21:51:02.375 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 10 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:51:02.375 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:51:02.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:51:02.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:51:02.390 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:02.385+0000 7f6d2089a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:51:02.391 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:02.385+0000 7f6d2089a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:51:02.392 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:02.389+0000 7f6d2089a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:51:02.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 10 2026-02-22T21:51:02.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:51:02.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=10 2026-02-22T21:51:02.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:51:02.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:51:02.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:02.575 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:51:02.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:51:02.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:02.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2026-02-22T21:51:02.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:03.772 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:03.772 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:03.772 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:51:03.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:51:03.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:03.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2026-02-22T21:51:03.898 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:03.894+0000 7f6d2089a8c0 -1 Falling back to public interface 2026-02-22T21:51:03.966 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:04.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:04.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:04.968 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:51:04.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:51:04.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:04.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2026-02-22T21:51:05.159 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:06.159 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:06.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:06.160 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:51:06.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:51:06.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:06.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2026-02-22T21:51:06.353 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:06.924 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:06.918+0000 7f6d2089a8c0 -1 osd.10 0 log_to_monitors true 2026-02-22T21:51:07.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:07.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:07.356 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:51:07.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:51:07.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:07.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2026-02-22T21:51:07.558 INFO:tasks.workunit.client.0.trial034.stdout:osd.10 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/146066069,v1:127.0.0.1:6883/146066069] [v2:127.0.0.1:6884/146066069,v1:127.0.0.1:6885/146066069] exists,up 186013bd-1dfa-4942-b1b1-68f394ee6968 2026-02-22T21:51:07.559 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:51:07.559 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:51:07.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:51:07.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2026-02-22T21:51:07.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T21:51:07.744 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' does not exist 2026-02-22T21:51:07.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2026-02-22T21:51:07.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T21:51:07.988 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T21:51:07.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:51:08.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2026-02-22T21:51:09.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2026-02-22T21:51:09.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:51:09.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:51:09.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:51:09.280 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:51:09.280 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:51:09.280 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:51:09.280 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:51:09.280 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:51:09.280 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:51:09.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:51:09.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:51:09.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:51:09.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:51:09.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:51:09.387 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:51:09.578 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:51:09.578 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:51:09.578 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:51:09.579 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:51:09.579 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:51:09.579 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:51:09.579 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:51:09.579 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:51:09.579 INFO:tasks.workunit.client.0.trial034.stderr:8 2026-02-22T21:51:09.579 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:51:09.579 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:51:09.579 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:51:09.579 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:09.580 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:51:09.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836501 2026-02-22T21:51:09.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836501 2026-02-22T21:51:09.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836501' 2026-02-22T21:51:09.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:09.656 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:51:09.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672980 2026-02-22T21:51:09.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672980 2026-02-22T21:51:09.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836501 1-42949672980' 2026-02-22T21:51:09.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:09.733 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:51:09.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=60129542161 2026-02-22T21:51:09.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 60129542161 2026-02-22T21:51:09.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836501 1-42949672980 2-60129542161' 2026-02-22T21:51:09.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:09.807 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:51:09.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411345 2026-02-22T21:51:09.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411345 2026-02-22T21:51:09.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836501 1-42949672980 2-60129542161 3-77309411345' 2026-02-22T21:51:09.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:09.882 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:51:09.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=94489280527 2026-02-22T21:51:09.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 94489280527 2026-02-22T21:51:09.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836501 1-42949672980 2-60129542161 3-77309411345 4-94489280527' 2026-02-22T21:51:09.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:09.958 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:51:10.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=111669149708 2026-02-22T21:51:10.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 111669149708 2026-02-22T21:51:10.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836501 1-42949672980 2-60129542161 3-77309411345 4-94489280527 5-111669149708' 2026-02-22T21:51:10.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:10.032 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:51:10.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=128849018890 2026-02-22T21:51:10.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 128849018890 2026-02-22T21:51:10.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836501 1-42949672980 2-60129542161 3-77309411345 4-94489280527 5-111669149708 6-128849018890' 2026-02-22T21:51:10.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:10.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T21:51:10.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=150323855368 2026-02-22T21:51:10.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 150323855368 2026-02-22T21:51:10.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836501 1-42949672980 2-60129542161 3-77309411345 4-94489280527 5-111669149708 6-128849018890 7-150323855368' 2026-02-22T21:51:10.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:10.182 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T21:51:10.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=171798691847 2026-02-22T21:51:10.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 171798691847 2026-02-22T21:51:10.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836501 1-42949672980 2-60129542161 3-77309411345 4-94489280527 5-111669149708 6-128849018890 7-150323855368 8-171798691847' 2026-02-22T21:51:10.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:10.259 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T21:51:10.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=193273528325 2026-02-22T21:51:10.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 193273528325 2026-02-22T21:51:10.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836501 1-42949672980 2-60129542161 3-77309411345 4-94489280527 5-111669149708 6-128849018890 7-150323855368 8-171798691847 9-193273528325' 2026-02-22T21:51:10.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:10.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2026-02-22T21:51:10.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=214748364803 2026-02-22T21:51:10.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 214748364803 2026-02-22T21:51:10.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836501 1-42949672980 2-60129542161 3-77309411345 4-94489280527 5-111669149708 6-128849018890 7-150323855368 8-171798691847 9-193273528325 10-214748364803' 2026-02-22T21:51:10.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:10.410 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836501 2026-02-22T21:51:10.410 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:10.411 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:51:10.412 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836501 2026-02-22T21:51:10.412 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:10.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836501 2026-02-22T21:51:10.414 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 21474836501 2026-02-22T21:51:10.414 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2026-02-22T21:51:10.414 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:51:10.614 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836501 -lt 21474836501 2026-02-22T21:51:10.614 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:10.615 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672980 2026-02-22T21:51:10.615 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:10.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:51:10.617 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672980 2026-02-22T21:51:10.618 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:10.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672980 2026-02-22T21:51:10.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2026-02-22T21:51:10.619 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 42949672980 2026-02-22T21:51:10.620 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:51:10.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672980 -lt 42949672980 2026-02-22T21:51:10.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:10.827 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-60129542161 2026-02-22T21:51:10.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:10.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:51:10.829 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-60129542161 2026-02-22T21:51:10.830 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:10.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=60129542161 2026-02-22T21:51:10.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 60129542161' 2026-02-22T21:51:10.831 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 60129542161 2026-02-22T21:51:10.832 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:51:11.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 60129542161 -lt 60129542161 2026-02-22T21:51:11.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:11.033 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-77309411345 2026-02-22T21:51:11.033 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:11.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:51:11.035 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-77309411345 2026-02-22T21:51:11.036 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:11.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411345 2026-02-22T21:51:11.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 77309411345' 2026-02-22T21:51:11.037 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 77309411345 2026-02-22T21:51:11.037 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:51:11.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411343 -lt 77309411345 2026-02-22T21:51:11.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:51:12.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:51:12.237 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:51:12.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411345 -lt 77309411345 2026-02-22T21:51:12.432 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:12.432 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-94489280527 2026-02-22T21:51:12.432 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:12.433 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:51:12.434 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-94489280527 2026-02-22T21:51:12.434 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:12.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=94489280527 2026-02-22T21:51:12.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 94489280527' 2026-02-22T21:51:12.436 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 94489280527 2026-02-22T21:51:12.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:51:12.630 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 94489280527 -lt 94489280527 2026-02-22T21:51:12.630 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:12.631 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-111669149708 2026-02-22T21:51:12.631 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:12.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:51:12.633 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-111669149708 2026-02-22T21:51:12.633 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:12.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=111669149708 2026-02-22T21:51:12.635 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 111669149708 2026-02-22T21:51:12.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 111669149708' 2026-02-22T21:51:12.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:51:12.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 111669149708 -lt 111669149708 2026-02-22T21:51:12.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:12.838 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-128849018890 2026-02-22T21:51:12.838 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:12.839 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:51:12.840 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-128849018890 2026-02-22T21:51:12.840 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:12.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=128849018890 2026-02-22T21:51:12.842 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 128849018890 2026-02-22T21:51:12.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 128849018890' 2026-02-22T21:51:12.842 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:51:13.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 128849018891 -lt 128849018890 2026-02-22T21:51:13.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:13.051 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 7-150323855368 2026-02-22T21:51:13.051 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:13.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=7 2026-02-22T21:51:13.053 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 7-150323855368 2026-02-22T21:51:13.053 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:13.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=150323855368 2026-02-22T21:51:13.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.7 seq 150323855368' 2026-02-22T21:51:13.055 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.7 seq 150323855368 2026-02-22T21:51:13.055 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T21:51:13.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 150323855369 -lt 150323855368 2026-02-22T21:51:13.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:13.245 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 8-171798691847 2026-02-22T21:51:13.245 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:13.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=8 2026-02-22T21:51:13.247 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 8-171798691847 2026-02-22T21:51:13.247 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:13.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=171798691847 2026-02-22T21:51:13.249 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.8 seq 171798691847 2026-02-22T21:51:13.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.8 seq 171798691847' 2026-02-22T21:51:13.249 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T21:51:13.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 171798691847 -lt 171798691847 2026-02-22T21:51:13.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:13.453 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 9-193273528325 2026-02-22T21:51:13.453 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:13.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=9 2026-02-22T21:51:13.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 9-193273528325 2026-02-22T21:51:13.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:13.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=193273528325 2026-02-22T21:51:13.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.9 seq 193273528325' 2026-02-22T21:51:13.457 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.9 seq 193273528325 2026-02-22T21:51:13.457 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T21:51:13.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 193273528325 -lt 193273528325 2026-02-22T21:51:13.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:13.658 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 10-214748364803 2026-02-22T21:51:13.658 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:13.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=10 2026-02-22T21:51:13.660 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 10-214748364803 2026-02-22T21:51:13.660 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:13.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=214748364803 2026-02-22T21:51:13.661 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.10 seq 214748364803 2026-02-22T21:51:13.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.10 seq 214748364803' 2026-02-22T21:51:13.662 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-22T21:51:13.857 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 214748364803 -lt 214748364803 2026-02-22T21:51:13.858 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:51:13.858 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:51:13.858 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:51:14.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 == 0 2026-02-22T21:51:14.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:51:14.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:51:14.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:51:14.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:51:14.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:51:14.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:51:14.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:51:14.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T21:51:14.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:51:14.298 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:51:14.298 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:51:14.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 4 2026-02-22T21:51:14.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:51:14.558 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:51:14.558 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2026-02-22T21:51:14.558 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:51:14.558 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:51:14.559 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:51:14.559 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:51:14.559 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:51:14.559 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.42545/ceph-osd.0.asok 2026-02-22T21:51:14.559 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2026-02-22T21:51:14.559 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.42545/ceph-osd.0.asok log flush 2026-02-22T21:51:14.621 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:51:14.627 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2026-02-22T21:51:14.628 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:49:34.403+0000 7f588bdd88c0 0 load: jerasure load: lrc 2026-02-22T21:51:14.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2026-02-22T21:51:14.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2026-02-22T21:51:14.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2026-02-22T21:51:14.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2026-02-22T21:51:14.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2026-02-22T21:51:15.244 INFO:tasks.workunit.client.0.trial034.stderr:pool 'ecpool' already exists 2026-02-22T21:51:15.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:51:16.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2026-02-22T21:51:16.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:51:16.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:51:16.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:51:16.257 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:51:16.258 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:51:16.258 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:51:16.258 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:51:16.258 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:51:16.258 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:51:16.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:51:16.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:51:16.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:51:16.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:51:16.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:51:16.355 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:51:16.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:51:16.554 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:51:16.554 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:51:16.555 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:51:16.555 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:51:16.555 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:51:16.555 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:51:16.555 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:51:16.555 INFO:tasks.workunit.client.0.trial034.stderr:8 2026-02-22T21:51:16.555 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:51:16.555 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:51:16.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:51:16.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:16.555 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:51:16.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836504 2026-02-22T21:51:16.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836504 2026-02-22T21:51:16.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836504' 2026-02-22T21:51:16.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:16.628 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:51:16.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672983 2026-02-22T21:51:16.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672983 2026-02-22T21:51:16.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836504 1-42949672983' 2026-02-22T21:51:16.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:16.702 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:51:16.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=60129542165 2026-02-22T21:51:16.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 60129542165 2026-02-22T21:51:16.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-60129542165' 2026-02-22T21:51:16.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:16.777 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:51:16.854 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411348 2026-02-22T21:51:16.854 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411348 2026-02-22T21:51:16.854 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-60129542165 3-77309411348' 2026-02-22T21:51:16.854 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:16.855 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:51:16.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=94489280530 2026-02-22T21:51:16.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 94489280530 2026-02-22T21:51:16.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-60129542165 3-77309411348 4-94489280530' 2026-02-22T21:51:16.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:16.930 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:51:17.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=111669149711 2026-02-22T21:51:17.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 111669149711 2026-02-22T21:51:17.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-60129542165 3-77309411348 4-94489280530 5-111669149711' 2026-02-22T21:51:17.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:17.008 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:51:17.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=128849018894 2026-02-22T21:51:17.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 128849018894 2026-02-22T21:51:17.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-60129542165 3-77309411348 4-94489280530 5-111669149711 6-128849018894' 2026-02-22T21:51:17.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:17.082 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T21:51:17.155 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=150323855372 2026-02-22T21:51:17.155 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 150323855372 2026-02-22T21:51:17.155 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-60129542165 3-77309411348 4-94489280530 5-111669149711 6-128849018894 7-150323855372' 2026-02-22T21:51:17.155 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:17.156 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T21:51:17.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=171798691850 2026-02-22T21:51:17.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 171798691850 2026-02-22T21:51:17.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-60129542165 3-77309411348 4-94489280530 5-111669149711 6-128849018894 7-150323855372 8-171798691850' 2026-02-22T21:51:17.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:17.232 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T21:51:17.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=193273528328 2026-02-22T21:51:17.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 193273528328 2026-02-22T21:51:17.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-60129542165 3-77309411348 4-94489280530 5-111669149711 6-128849018894 7-150323855372 8-171798691850 9-193273528328' 2026-02-22T21:51:17.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:17.306 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2026-02-22T21:51:17.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=214748364806 2026-02-22T21:51:17.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 214748364806 2026-02-22T21:51:17.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-60129542165 3-77309411348 4-94489280530 5-111669149711 6-128849018894 7-150323855372 8-171798691850 9-193273528328 10-214748364806' 2026-02-22T21:51:17.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:17.382 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836504 2026-02-22T21:51:17.383 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:17.383 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:51:17.384 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836504 2026-02-22T21:51:17.384 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:17.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836504 2026-02-22T21:51:17.385 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 21474836504 2026-02-22T21:51:17.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2026-02-22T21:51:17.386 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:51:17.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836502 -lt 21474836504 2026-02-22T21:51:17.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:51:18.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:51:18.585 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:51:18.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836504 -lt 21474836504 2026-02-22T21:51:18.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:18.784 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672983 2026-02-22T21:51:18.785 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:18.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:51:18.786 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672983 2026-02-22T21:51:18.787 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:18.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672983 2026-02-22T21:51:18.788 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 42949672983 2026-02-22T21:51:18.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2026-02-22T21:51:18.789 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:51:18.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672984 -lt 42949672983 2026-02-22T21:51:18.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:18.981 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-60129542165 2026-02-22T21:51:18.981 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:18.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:51:18.983 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-60129542165 2026-02-22T21:51:18.983 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:18.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=60129542165 2026-02-22T21:51:18.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 60129542165' 2026-02-22T21:51:18.985 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 60129542165 2026-02-22T21:51:18.985 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:51:19.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 60129542165 -lt 60129542165 2026-02-22T21:51:19.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:19.188 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-77309411348 2026-02-22T21:51:19.188 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:19.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:51:19.190 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-77309411348 2026-02-22T21:51:19.191 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:19.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411348 2026-02-22T21:51:19.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 77309411348' 2026-02-22T21:51:19.192 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 77309411348 2026-02-22T21:51:19.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:51:19.389 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411348 -lt 77309411348 2026-02-22T21:51:19.389 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:19.390 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-94489280530 2026-02-22T21:51:19.390 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:19.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:51:19.392 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-94489280530 2026-02-22T21:51:19.392 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:19.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=94489280530 2026-02-22T21:51:19.393 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 94489280530 2026-02-22T21:51:19.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 94489280530' 2026-02-22T21:51:19.394 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:51:19.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 94489280530 -lt 94489280530 2026-02-22T21:51:19.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:19.591 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-111669149711 2026-02-22T21:51:19.591 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:19.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:51:19.593 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-111669149711 2026-02-22T21:51:19.593 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:19.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=111669149711 2026-02-22T21:51:19.595 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 111669149711 2026-02-22T21:51:19.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 111669149711' 2026-02-22T21:51:19.595 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:51:19.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 111669149712 -lt 111669149711 2026-02-22T21:51:19.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:19.796 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-128849018894 2026-02-22T21:51:19.796 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:19.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:51:19.798 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-128849018894 2026-02-22T21:51:19.798 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:19.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=128849018894 2026-02-22T21:51:19.799 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 128849018894 2026-02-22T21:51:19.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 128849018894' 2026-02-22T21:51:19.800 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:51:20.000 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 128849018894 -lt 128849018894 2026-02-22T21:51:20.000 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:20.000 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 7-150323855372 2026-02-22T21:51:20.000 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:20.001 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=7 2026-02-22T21:51:20.003 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 7-150323855372 2026-02-22T21:51:20.003 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:20.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=150323855372 2026-02-22T21:51:20.004 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.7 seq 150323855372 2026-02-22T21:51:20.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.7 seq 150323855372' 2026-02-22T21:51:20.005 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T21:51:20.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 150323855372 -lt 150323855372 2026-02-22T21:51:20.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:20.203 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 8-171798691850 2026-02-22T21:51:20.203 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:20.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=8 2026-02-22T21:51:20.205 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 8-171798691850 2026-02-22T21:51:20.205 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:20.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=171798691850 2026-02-22T21:51:20.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.8 seq 171798691850' 2026-02-22T21:51:20.206 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.8 seq 171798691850 2026-02-22T21:51:20.207 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T21:51:20.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 171798691850 -lt 171798691850 2026-02-22T21:51:20.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:20.409 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 9-193273528328 2026-02-22T21:51:20.409 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:20.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=9 2026-02-22T21:51:20.411 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 9-193273528328 2026-02-22T21:51:20.411 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:20.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=193273528328 2026-02-22T21:51:20.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.9 seq 193273528328' 2026-02-22T21:51:20.413 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.9 seq 193273528328 2026-02-22T21:51:20.413 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T21:51:20.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 193273528329 -lt 193273528328 2026-02-22T21:51:20.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:20.612 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 10-214748364806 2026-02-22T21:51:20.612 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:20.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=10 2026-02-22T21:51:20.614 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 10-214748364806 2026-02-22T21:51:20.614 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:20.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=214748364806 2026-02-22T21:51:20.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.10 seq 214748364806' 2026-02-22T21:51:20.616 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.10 seq 214748364806 2026-02-22T21:51:20.616 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-22T21:51:20.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 214748364807 -lt 214748364806 2026-02-22T21:51:20.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:51:20.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:51:20.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:51:21.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 16 == 0 2026-02-22T21:51:21.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:51:21.075 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:51:21.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:51:21.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:51:21.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:51:21.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:51:21.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:51:21.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=16 2026-02-22T21:51:21.266 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:51:21.266 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:51:21.266 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:51:21.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 16 = 16 2026-02-22T21:51:21.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:51:21.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:51:21.520 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2026-02-22T21:51:21.520 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T21:51:21.522 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2026-02-22T21:51:21.522 INFO:tasks.workunit.client.0.trial034.stderr:TEST_chunk_mapping 2026-02-22T21:51:21.522 INFO:tasks.workunit.client.0.trial034.stderr:TEST_rados_put_get_isa 2026-02-22T21:51:21.522 INFO:tasks.workunit.client.0.trial034.stderr:TEST_rados_put_get_jerasure 2026-02-22T21:51:21.522 INFO:tasks.workunit.client.0.trial034.stderr:TEST_rados_put_get_lrc_advanced 2026-02-22T21:51:21.522 INFO:tasks.workunit.client.0.trial034.stderr:TEST_rados_put_get_lrc_kml 2026-02-22T21:51:21.522 INFO:tasks.workunit.client.0.trial034.stderr:TEST_rados_put_get_shec' 2026-02-22T21:51:21.523 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-22T21:51:21.523 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2026-02-22T21:51:21.523 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: TEST_alignment_constraints: local payload=ABC 2026-02-22T21:51:21.523 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: TEST_alignment_constraints: echo ABC 2026-02-22T21:51:21.523 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-22T21:51:21.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: local stripe_unit=4096 2026-02-22T21:51:21.537 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2026-02-22T21:51:21.537 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: grep k= 2026-02-22T21:51:21.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: eval local k=2 2026-02-22T21:51:21.738 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: local k=2 2026-02-22T21:51:21.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: TEST_alignment_constraints: local block_size=8191 2026-02-22T21:51:21.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:263: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2026-02-22T21:51:21.739 INFO:tasks.workunit.client.0.trial034.stderr:2+0 records in 2026-02-22T21:51:21.739 INFO:tasks.workunit.client.0.trial034.stderr:2+0 records out 2026-02-22T21:51:21.739 INFO:tasks.workunit.client.0.trial034.stderr:16382 bytes (16 kB, 16 KiB) copied, 9.9328e-05 s, 165 MB/s 2026-02-22T21:51:21.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:264: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2026-02-22T21:51:21.762 INFO:tasks.workunit.client.0.trial034.stderr:INFO: op_size has been rounded to 8192 2026-02-22T21:51:21.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2026-02-22T21:51:21.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-22T21:51:21.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2026-02-22T21:51:21.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: TEST_chunk_mapping: local dir=td/test-erasure-code 2026-02-22T21:51:21.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:311: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2026-02-22T21:51:21.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2026-02-22T21:51:21.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=ecpool 2026-02-22T21:51:21.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=0 2026-02-22T21:51:21.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=1 2026-02-22T21:51:21.770 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2026-02-22T21:51:21.770 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-22T21:51:21.778 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2026-02-22T21:51:21.778 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2026-02-22T21:51:21.779 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2026-02-22T21:51:21.779 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-22T21:51:21.787 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2026-02-22T21:51:21.787 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2026-02-22T21:51:21.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2026-02-22T21:51:21.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2026-02-22T21:51:21.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2026-02-22T21:51:21.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2026-02-22T21:51:21.836 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2026-02-22T21:51:21.836 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2026-02-22T21:51:21.836 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2026-02-22T21:51:21.836 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2026-02-22T21:51:21.837 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:51:22.029 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T21:51:22.029 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:51:22.029 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:51:22.030 INFO:tasks.workunit.client.0.trial034.stderr:0' 2026-02-22T21:51:22.030 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2026-02-22T21:51:22.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('3' '5' '9' '0') 2026-02-22T21:51:22.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2026-02-22T21:51:22.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2026-02-22T21:51:22.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2026-02-22T21:51:22.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2026-02-22T21:51:22.094 INFO:tasks.workunit.client.0.trial034.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-22T21:51:22.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2026-02-22T21:51:22.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2026-02-22T21:51:22.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2026-02-22T21:51:22.168 INFO:tasks.workunit.client.0.trial034.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-22T21:51:22.170 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2026-02-22T21:51:22.170 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2026-02-22T21:51:22.170 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2026-02-22T21:51:22.239 INFO:tasks.workunit.client.0.trial034.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-22T21:51:22.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2026-02-22T21:51:22.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2026-02-22T21:51:22.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2026-02-22T21:51:22.304 INFO:tasks.workunit.client.0.trial034.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-22T21:51:22.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2026-02-22T21:51:22.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2026-02-22T21:51:22.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:51:22.307 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2026-02-22T21:51:22.309 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2026-02-22T21:51:22.309 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2026-02-22T21:51:22.309 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2026-02-22T21:51:22.310 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2026-02-22T21:51:22.310 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:51:22.509 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T21:51:22.509 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:51:22.509 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:51:22.509 INFO:tasks.workunit.client.0.trial034.stderr:0' 2026-02-22T21:51:22.509 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2026-02-22T21:51:22.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('3' '5' '9' '0') 2026-02-22T21:51:22.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2026-02-22T21:51:22.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2026-02-22T21:51:22.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2026-02-22T21:51:22.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T21:51:22.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2026-02-22T21:51:22.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T21:51:22.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2026-02-22T21:51:22.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2026-02-22T21:51:22.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T21:51:22.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2026-02-22T21:51:22.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T21:51:22.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2026-02-22T21:51:22.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTecpool 2026-02-22T21:51:22.512 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:51:22.512 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:51:22.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:51:22.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:51:22.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:51:23.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:51:23.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2026-02-22T21:51:23.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2026-02-22T21:51:23.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T21:51:23.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2026-02-22T21:51:23.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T21:51:23.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2026-02-22T21:51:23.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2026-02-22T21:51:24.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2026-02-22T21:51:24.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2026-02-22T21:51:24.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T21:51:24.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2026-02-22T21:51:24.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/3 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:51:24.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:51:24.698 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:51:24.698 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:51:24.698 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:51:24.698 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:51:24.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:51:24.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:51:24.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/3 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr:start osd.3 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/3/whoami 2026-02-22T21:51:24.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2026-02-22T21:51:24.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T21:51:24.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T21:51:24.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T21:51:24.715 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:24.710+0000 7f187878f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:51:24.717 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:24.714+0000 7f187878f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:51:24.718 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:24.714+0000 7f187878f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:51:24.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2026-02-22T21:51:24.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:51:24.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T21:51:24.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:51:24.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:51:24.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:24.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:51:24.896 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:51:24.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:24.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:51:25.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:26.021 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:26.018+0000 7f187878f8c0 -1 Falling back to public interface 2026-02-22T21:51:26.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:26.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:26.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:51:26.092 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:51:26.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:26.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:51:26.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:27.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:27.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:27.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:51:27.294 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:51:27.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:27.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:51:27.492 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:28.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:28.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:28.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:51:28.493 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:51:28.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:28.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:51:28.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:29.048 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:29.046+0000 7f187878f8c0 -1 osd.3 64 log_to_monitors true 2026-02-22T21:51:29.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:29.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:29.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:51:29.693 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:51:29.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:29.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:51:29.890 INFO:tasks.workunit.client.0.trial034.stderr:osd.3 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [18,64) [v2:127.0.0.1:6826/3315610006,v1:127.0.0.1:6827/3315610006] [v2:127.0.0.1:6828/3315610006,v1:127.0.0.1:6829/3315610006] exists,up 5c6cf897-2094-44c0-8256-2f77e5a5fcf4 2026-02-22T21:51:29.890 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:51:29.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:51:29.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:51:29.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T21:51:29.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:51:29.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:51:29.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:51:29.892 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:51:29.892 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:51:29.892 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:51:29.892 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:51:29.892 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:51:29.892 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:51:29.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:51:29.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:51:29.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:51:29.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:51:29.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:51:29.998 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:51:30.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr:8 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:30.192 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:51:30.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836509 2026-02-22T21:51:30.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836509 2026-02-22T21:51:30.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836509' 2026-02-22T21:51:30.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:30.269 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:51:30.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672988 2026-02-22T21:51:30.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672988 2026-02-22T21:51:30.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836509 1-42949672988' 2026-02-22T21:51:30.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:30.345 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:51:30.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=60129542169 2026-02-22T21:51:30.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 60129542169 2026-02-22T21:51:30.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-60129542169' 2026-02-22T21:51:30.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:30.419 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:51:30.492 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=292057776131 2026-02-22T21:51:30.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 292057776131 2026-02-22T21:51:30.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-60129542169 3-292057776131' 2026-02-22T21:51:30.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:30.493 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:51:30.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=94489280535 2026-02-22T21:51:30.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 94489280535 2026-02-22T21:51:30.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-60129542169 3-292057776131 4-94489280535' 2026-02-22T21:51:30.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:30.569 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:51:30.644 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=111669149716 2026-02-22T21:51:30.644 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 111669149716 2026-02-22T21:51:30.645 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-60129542169 3-292057776131 4-94489280535 5-111669149716' 2026-02-22T21:51:30.645 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:30.645 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:51:30.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=128849018898 2026-02-22T21:51:30.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 128849018898 2026-02-22T21:51:30.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-60129542169 3-292057776131 4-94489280535 5-111669149716 6-128849018898' 2026-02-22T21:51:30.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:30.719 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T21:51:30.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=150323855377 2026-02-22T21:51:30.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 150323855377 2026-02-22T21:51:30.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-60129542169 3-292057776131 4-94489280535 5-111669149716 6-128849018898 7-150323855377' 2026-02-22T21:51:30.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:30.793 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T21:51:30.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=171798691855 2026-02-22T21:51:30.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 171798691855 2026-02-22T21:51:30.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-60129542169 3-292057776131 4-94489280535 5-111669149716 6-128849018898 7-150323855377 8-171798691855' 2026-02-22T21:51:30.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:30.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T21:51:30.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=193273528333 2026-02-22T21:51:30.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 193273528333 2026-02-22T21:51:30.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-60129542169 3-292057776131 4-94489280535 5-111669149716 6-128849018898 7-150323855377 8-171798691855 9-193273528333' 2026-02-22T21:51:30.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:30.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2026-02-22T21:51:31.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=214748364811 2026-02-22T21:51:31.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 214748364811 2026-02-22T21:51:31.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-60129542169 3-292057776131 4-94489280535 5-111669149716 6-128849018898 7-150323855377 8-171798691855 9-193273528333 10-214748364811' 2026-02-22T21:51:31.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:31.023 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836509 2026-02-22T21:51:31.023 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:31.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:51:31.025 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836509 2026-02-22T21:51:31.025 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:31.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836509 2026-02-22T21:51:31.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2026-02-22T21:51:31.026 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 21474836509 2026-02-22T21:51:31.026 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:51:31.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836507 -lt 21474836509 2026-02-22T21:51:31.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:51:32.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:51:32.229 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:51:32.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836509 -lt 21474836509 2026-02-22T21:51:32.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:32.421 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672988 2026-02-22T21:51:32.422 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:32.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:51:32.423 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672988 2026-02-22T21:51:32.423 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:32.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672988 2026-02-22T21:51:32.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2026-02-22T21:51:32.425 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 42949672988 2026-02-22T21:51:32.425 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:51:32.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672988 -lt 42949672988 2026-02-22T21:51:32.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:32.624 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-60129542169 2026-02-22T21:51:32.624 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:32.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:51:32.625 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-60129542169 2026-02-22T21:51:32.625 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:32.627 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=60129542169 2026-02-22T21:51:32.627 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 60129542169' 2026-02-22T21:51:32.627 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 60129542169 2026-02-22T21:51:32.627 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:51:32.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 60129542170 -lt 60129542169 2026-02-22T21:51:32.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:32.832 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-292057776131 2026-02-22T21:51:32.832 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:32.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:51:32.834 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-292057776131 2026-02-22T21:51:32.834 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:32.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=292057776131 2026-02-22T21:51:32.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 292057776131' 2026-02-22T21:51:32.835 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 292057776131 2026-02-22T21:51:32.835 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:51:33.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 292057776131 -lt 292057776131 2026-02-22T21:51:33.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:33.040 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-94489280535 2026-02-22T21:51:33.040 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:33.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:51:33.042 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-94489280535 2026-02-22T21:51:33.042 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:33.043 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=94489280535 2026-02-22T21:51:33.044 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 94489280535' 2026-02-22T21:51:33.044 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.4 seq 94489280535 2026-02-22T21:51:33.044 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:51:33.239 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 94489280535 -lt 94489280535 2026-02-22T21:51:33.239 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:33.240 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-111669149716 2026-02-22T21:51:33.240 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:33.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:51:33.241 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-111669149716 2026-02-22T21:51:33.241 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:33.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=111669149716 2026-02-22T21:51:33.243 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 111669149716' 2026-02-22T21:51:33.243 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.5 seq 111669149716 2026-02-22T21:51:33.243 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:51:33.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 111669149716 -lt 111669149716 2026-02-22T21:51:33.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:33.444 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-128849018898 2026-02-22T21:51:33.444 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:33.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:51:33.446 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-128849018898 2026-02-22T21:51:33.446 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:33.447 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=128849018898 2026-02-22T21:51:33.447 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 128849018898' 2026-02-22T21:51:33.447 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.6 seq 128849018898 2026-02-22T21:51:33.448 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:51:33.644 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 128849018899 -lt 128849018898 2026-02-22T21:51:33.644 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:33.644 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 7-150323855377 2026-02-22T21:51:33.645 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:33.646 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=7 2026-02-22T21:51:33.646 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 7-150323855377 2026-02-22T21:51:33.647 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:33.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=150323855377 2026-02-22T21:51:33.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.7 seq 150323855377' 2026-02-22T21:51:33.648 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.7 seq 150323855377 2026-02-22T21:51:33.648 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T21:51:33.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 150323855377 -lt 150323855377 2026-02-22T21:51:33.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:33.845 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 8-171798691855 2026-02-22T21:51:33.845 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:33.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=8 2026-02-22T21:51:33.846 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 8-171798691855 2026-02-22T21:51:33.846 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:33.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=171798691855 2026-02-22T21:51:33.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.8 seq 171798691855' 2026-02-22T21:51:33.848 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.8 seq 171798691855 2026-02-22T21:51:33.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T21:51:34.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 171798691855 -lt 171798691855 2026-02-22T21:51:34.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:34.048 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 9-193273528333 2026-02-22T21:51:34.048 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:34.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=9 2026-02-22T21:51:34.049 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 9-193273528333 2026-02-22T21:51:34.049 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:34.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=193273528333 2026-02-22T21:51:34.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.9 seq 193273528333' 2026-02-22T21:51:34.051 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.9 seq 193273528333 2026-02-22T21:51:34.051 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T21:51:34.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 193273528333 -lt 193273528333 2026-02-22T21:51:34.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:34.260 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 10-214748364811 2026-02-22T21:51:34.260 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:34.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=10 2026-02-22T21:51:34.261 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 10-214748364811 2026-02-22T21:51:34.261 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:34.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=214748364811 2026-02-22T21:51:34.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.10 seq 214748364811' 2026-02-22T21:51:34.263 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.10 seq 214748364811 2026-02-22T21:51:34.263 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-22T21:51:34.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 214748364812 -lt 214748364811 2026-02-22T21:51:34.465 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:51:34.465 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:51:34.465 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:51:34.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 16 == 0 2026-02-22T21:51:34.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:51:34.717 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:51:34.717 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:51:34.717 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:51:34.717 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:51:34.717 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:51:34.717 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:51:34.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=16 2026-02-22T21:51:34.910 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:51:34.910 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:51:34.910 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:51:35.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 16 = 16 2026-02-22T21:51:35.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:51:35.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:51:35.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2026-02-22T21:51:35.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2026-02-22T21:51:35.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T21:51:35.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2026-02-22T21:51:35.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T21:51:35.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2026-02-22T21:51:35.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2026-02-22T21:51:35.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T21:51:35.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2026-02-22T21:51:35.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T21:51:35.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2026-02-22T21:51:35.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDecpool 2026-02-22T21:51:35.164 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:51:35.165 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:51:35.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:51:35.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:51:35.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:51:36.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:51:36.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2026-02-22T21:51:36.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2026-02-22T21:51:36.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T21:51:36.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2026-02-22T21:51:36.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T21:51:36.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2026-02-22T21:51:36.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2026-02-22T21:51:37.330 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2026-02-22T21:51:37.331 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2026-02-22T21:51:37.331 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T21:51:37.331 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2026-02-22T21:51:37.331 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T21:51:37.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/5 2026-02-22T21:51:37.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:51:37.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:51:37.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:51:37.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:51:37.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2026-02-22T21:51:37.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2026-02-22T21:51:37.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:51:37.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T21:51:37.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:51:37.333 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T21:51:37.333 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:51:37.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:51:37.334 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:51:37.334 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:51:37.334 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:51:37.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:51:37.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:51:37.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:51:37.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:51:37.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:51:37.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:51:37.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:51:37.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:51:37.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:51:37.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T21:51:37.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T21:51:37.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/5 2026-02-22T21:51:37.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2026-02-22T21:51:37.336 INFO:tasks.workunit.client.0.trial034.stderr:start osd.5 2026-02-22T21:51:37.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:51:37.337 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/5/whoami 2026-02-22T21:51:37.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2026-02-22T21:51:37.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T21:51:37.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T21:51:37.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T21:51:37.349 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:37.346+0000 7fa48c8b58c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:51:37.350 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:37.346+0000 7fa48c8b58c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:51:37.351 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:37.346+0000 7fa48c8b58c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:51:37.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2026-02-22T21:51:37.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:51:37.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2026-02-22T21:51:37.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:51:37.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:51:37.531 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:37.531 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:51:37.531 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:51:37.531 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:37.531 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:51:37.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:38.613 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:38.610+0000 7fa48c8b58c0 -1 Falling back to public interface 2026-02-22T21:51:38.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:38.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:38.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:51:38.730 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:51:38.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:38.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:51:38.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:39.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:39.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:39.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:51:39.922 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:51:39.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:39.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:51:40.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:41.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:41.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:41.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:51:41.120 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:51:41.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:41.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:51:41.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:41.860 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:41.858+0000 7fa48c8b58c0 -1 osd.5 71 log_to_monitors true 2026-02-22T21:51:42.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:42.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:42.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:51:42.313 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:51:42.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:42.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:51:42.509 INFO:tasks.workunit.client.0.trial034.stderr:osd.5 up in weight 1 up_from 75 up_thru 75 down_at 72 last_clean_interval [26,71) [v2:127.0.0.1:6842/4254400672,v1:127.0.0.1:6843/4254400672] [v2:127.0.0.1:6844/4254400672,v1:127.0.0.1:6845/4254400672] exists,up c21beadb-f200-42f4-8059-76a7ad8f17d4 2026-02-22T21:51:42.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:51:42.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:51:42.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:51:42.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T21:51:42.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:51:42.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:51:42.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:51:42.510 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:51:42.510 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:51:42.510 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:51:42.510 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:51:42.510 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:51:42.510 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:51:42.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:51:42.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:51:42.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:51:42.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:51:42.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:51:42.613 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:51:42.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:51:42.806 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:51:42.806 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:51:42.806 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:51:42.806 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:51:42.806 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:51:42.806 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:51:42.806 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:51:42.806 INFO:tasks.workunit.client.0.trial034.stderr:8 2026-02-22T21:51:42.806 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:51:42.807 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:51:42.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:51:42.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:42.807 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:51:42.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836513 2026-02-22T21:51:42.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836513 2026-02-22T21:51:42.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836513' 2026-02-22T21:51:42.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:42.882 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:51:42.958 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672993 2026-02-22T21:51:42.958 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672993 2026-02-22T21:51:42.958 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836513 1-42949672993' 2026-02-22T21:51:42.958 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:42.959 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:51:43.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=60129542174 2026-02-22T21:51:43.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 60129542174 2026-02-22T21:51:43.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542174' 2026-02-22T21:51:43.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:43.032 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:51:43.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=292057776135 2026-02-22T21:51:43.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 292057776135 2026-02-22T21:51:43.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542174 3-292057776135' 2026-02-22T21:51:43.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:43.106 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:51:43.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=94489280539 2026-02-22T21:51:43.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 94489280539 2026-02-22T21:51:43.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542174 3-292057776135 4-94489280539' 2026-02-22T21:51:43.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:43.182 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:51:43.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=322122547203 2026-02-22T21:51:43.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 322122547203 2026-02-22T21:51:43.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542174 3-292057776135 4-94489280539 5-322122547203' 2026-02-22T21:51:43.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:43.258 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:51:43.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=128849018903 2026-02-22T21:51:43.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 128849018903 2026-02-22T21:51:43.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542174 3-292057776135 4-94489280539 5-322122547203 6-128849018903' 2026-02-22T21:51:43.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:43.336 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T21:51:43.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=150323855381 2026-02-22T21:51:43.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 150323855381 2026-02-22T21:51:43.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542174 3-292057776135 4-94489280539 5-322122547203 6-128849018903 7-150323855381' 2026-02-22T21:51:43.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:43.409 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T21:51:43.484 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=171798691859 2026-02-22T21:51:43.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 171798691859 2026-02-22T21:51:43.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542174 3-292057776135 4-94489280539 5-322122547203 6-128849018903 7-150323855381 8-171798691859' 2026-02-22T21:51:43.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:43.485 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T21:51:43.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=193273528337 2026-02-22T21:51:43.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 193273528337 2026-02-22T21:51:43.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542174 3-292057776135 4-94489280539 5-322122547203 6-128849018903 7-150323855381 8-171798691859 9-193273528337' 2026-02-22T21:51:43.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:43.560 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2026-02-22T21:51:43.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=214748364816 2026-02-22T21:51:43.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 214748364816 2026-02-22T21:51:43.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542174 3-292057776135 4-94489280539 5-322122547203 6-128849018903 7-150323855381 8-171798691859 9-193273528337 10-214748364816' 2026-02-22T21:51:43.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:43.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836513 2026-02-22T21:51:43.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:43.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:51:43.636 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836513 2026-02-22T21:51:43.637 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:43.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836513 2026-02-22T21:51:43.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2026-02-22T21:51:43.638 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 21474836513 2026-02-22T21:51:43.638 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:51:43.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836511 -lt 21474836513 2026-02-22T21:51:43.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:51:44.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:51:44.830 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:51:45.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836514 -lt 21474836513 2026-02-22T21:51:45.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:45.027 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672993 2026-02-22T21:51:45.027 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:45.028 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:51:45.029 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672993 2026-02-22T21:51:45.029 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:45.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672993 2026-02-22T21:51:45.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2026-02-22T21:51:45.030 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 42949672993 2026-02-22T21:51:45.030 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:51:45.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672993 -lt 42949672993 2026-02-22T21:51:45.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:45.233 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-60129542174 2026-02-22T21:51:45.233 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:45.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:51:45.234 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-60129542174 2026-02-22T21:51:45.235 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:45.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=60129542174 2026-02-22T21:51:45.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 60129542174' 2026-02-22T21:51:45.236 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 60129542174 2026-02-22T21:51:45.236 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:51:45.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 60129542174 -lt 60129542174 2026-02-22T21:51:45.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:45.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-292057776135 2026-02-22T21:51:45.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:45.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:51:45.437 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-292057776135 2026-02-22T21:51:45.437 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:45.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=292057776135 2026-02-22T21:51:45.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 292057776135' 2026-02-22T21:51:45.439 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 292057776135 2026-02-22T21:51:45.439 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:51:45.644 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 292057776135 -lt 292057776135 2026-02-22T21:51:45.645 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:45.645 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-94489280539 2026-02-22T21:51:45.646 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:45.646 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:51:45.647 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-94489280539 2026-02-22T21:51:45.647 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:45.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=94489280539 2026-02-22T21:51:45.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 94489280539' 2026-02-22T21:51:45.648 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.4 seq 94489280539 2026-02-22T21:51:45.649 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:51:45.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 94489280540 -lt 94489280539 2026-02-22T21:51:45.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:45.850 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-322122547203 2026-02-22T21:51:45.850 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:45.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:51:45.852 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-322122547203 2026-02-22T21:51:45.852 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:45.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=322122547203 2026-02-22T21:51:45.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 322122547203' 2026-02-22T21:51:45.854 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.5 seq 322122547203 2026-02-22T21:51:45.854 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:51:46.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 322122547203 -lt 322122547203 2026-02-22T21:51:46.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:46.061 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-128849018903 2026-02-22T21:51:46.061 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:46.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:51:46.063 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-128849018903 2026-02-22T21:51:46.063 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:46.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=128849018903 2026-02-22T21:51:46.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 128849018903' 2026-02-22T21:51:46.065 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.6 seq 128849018903 2026-02-22T21:51:46.065 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:51:46.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 128849018903 -lt 128849018903 2026-02-22T21:51:46.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:46.262 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 7-150323855381 2026-02-22T21:51:46.262 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:46.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=7 2026-02-22T21:51:46.264 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 7-150323855381 2026-02-22T21:51:46.264 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:46.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=150323855381 2026-02-22T21:51:46.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.7 seq 150323855381' 2026-02-22T21:51:46.266 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.7 seq 150323855381 2026-02-22T21:51:46.266 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T21:51:46.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 150323855382 -lt 150323855381 2026-02-22T21:51:46.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:46.463 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 8-171798691859 2026-02-22T21:51:46.463 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:46.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=8 2026-02-22T21:51:46.465 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 8-171798691859 2026-02-22T21:51:46.465 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:46.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=171798691859 2026-02-22T21:51:46.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.8 seq 171798691859' 2026-02-22T21:51:46.466 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.8 seq 171798691859 2026-02-22T21:51:46.466 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T21:51:46.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 171798691860 -lt 171798691859 2026-02-22T21:51:46.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:46.665 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 9-193273528337 2026-02-22T21:51:46.665 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:46.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=9 2026-02-22T21:51:46.667 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 9-193273528337 2026-02-22T21:51:46.667 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:46.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=193273528337 2026-02-22T21:51:46.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.9 seq 193273528337' 2026-02-22T21:51:46.669 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.9 seq 193273528337 2026-02-22T21:51:46.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T21:51:46.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 193273528338 -lt 193273528337 2026-02-22T21:51:46.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:46.876 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 10-214748364816 2026-02-22T21:51:46.876 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:46.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=10 2026-02-22T21:51:46.878 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 10-214748364816 2026-02-22T21:51:46.878 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:46.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=214748364816 2026-02-22T21:51:46.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.10 seq 214748364816' 2026-02-22T21:51:46.880 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.10 seq 214748364816 2026-02-22T21:51:46.880 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-22T21:51:47.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 214748364816 -lt 214748364816 2026-02-22T21:51:47.073 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:51:47.074 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:51:47.074 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:51:47.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 16 == 0 2026-02-22T21:51:47.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:51:47.333 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:51:47.333 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:51:47.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:51:47.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:51:47.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:51:47.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:51:47.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=16 2026-02-22T21:51:47.525 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:51:47.525 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:51:47.525 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:51:47.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 16 = 16 2026-02-22T21:51:47.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:51:47.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:51:47.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:313: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2026-02-22T21:51:48.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:318: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2026-02-22T21:51:48.215 INFO:tasks.workunit.client.0.trial034.stdout:crush-device-class= 2026-02-22T21:51:48.216 INFO:tasks.workunit.client.0.trial034.stdout:crush-failure-domain=host 2026-02-22T21:51:48.216 INFO:tasks.workunit.client.0.trial034.stdout:crush-num-failure-domains=0 2026-02-22T21:51:48.216 INFO:tasks.workunit.client.0.trial034.stdout:crush-osds-per-failure-domain=0 2026-02-22T21:51:48.216 INFO:tasks.workunit.client.0.trial034.stdout:crush-root=default 2026-02-22T21:51:48.216 INFO:tasks.workunit.client.0.trial034.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2026-02-22T21:51:48.216 INFO:tasks.workunit.client.0.trial034.stdout:k=-1 2026-02-22T21:51:48.216 INFO:tasks.workunit.client.0.trial034.stdout:l=-1 2026-02-22T21:51:48.216 INFO:tasks.workunit.client.0.trial034.stdout:layers=[ [ "cDD", "" ] ] 2026-02-22T21:51:48.217 INFO:tasks.workunit.client.0.trial034.stdout:m=-1 2026-02-22T21:51:48.217 INFO:tasks.workunit.client.0.trial034.stdout:mapping=_DD 2026-02-22T21:51:48.217 INFO:tasks.workunit.client.0.trial034.stdout:plugin=lrc 2026-02-22T21:51:48.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:319: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2026-02-22T21:51:48.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2026-02-22T21:51:48.580 INFO:tasks.workunit.client.0.trial034.stderr:pool 'remap-pool' already exists 2026-02-22T21:51:48.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:51:49.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:327: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2026-02-22T21:51:49.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2026-02-22T21:51:49.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=remap-pool 2026-02-22T21:51:49.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=1 2026-02-22T21:51:49.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=2 2026-02-22T21:51:49.593 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2026-02-22T21:51:49.593 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-22T21:51:49.604 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2026-02-22T21:51:49.604 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2026-02-22T21:51:49.605 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2026-02-22T21:51:49.605 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2026-02-22T21:51:49.614 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2026-02-22T21:51:49.614 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2026-02-22T21:51:49.614 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2026-02-22T21:51:49.614 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2026-02-22T21:51:49.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2026-02-22T21:51:49.644 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2026-02-22T21:51:49.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2026-02-22T21:51:49.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2026-02-22T21:51:49.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2026-02-22T21:51:49.670 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2026-02-22T21:51:49.670 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:51:49.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2026-02-22T21:51:49.870 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:51:49.870 INFO:tasks.workunit.client.0.trial034.stderr:9' 2026-02-22T21:51:49.870 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2026-02-22T21:51:49.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('4' '6' '9') 2026-02-22T21:51:49.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2026-02-22T21:51:49.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2026-02-22T21:51:49.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2026-02-22T21:51:49.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2026-02-22T21:51:49.932 INFO:tasks.workunit.client.0.trial034.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-22T21:51:49.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2026-02-22T21:51:49.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2026-02-22T21:51:49.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2026-02-22T21:51:50.007 INFO:tasks.workunit.client.0.trial034.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-22T21:51:50.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2026-02-22T21:51:50.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2026-02-22T21:51:50.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2026-02-22T21:51:50.071 INFO:tasks.workunit.client.0.trial034.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2026-02-22T21:51:50.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2026-02-22T21:51:50.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2026-02-22T21:51:50.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:51:50.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2026-02-22T21:51:50.075 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2026-02-22T21:51:50.075 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2026-02-22T21:51:50.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2026-02-22T21:51:50.076 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2026-02-22T21:51:50.076 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:51:50.280 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2026-02-22T21:51:50.281 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:51:50.281 INFO:tasks.workunit.client.0.trial034.stderr:9' 2026-02-22T21:51:50.281 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2026-02-22T21:51:50.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('4' '6' '9') 2026-02-22T21:51:50.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2026-02-22T21:51:50.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2026-02-22T21:51:50.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2026-02-22T21:51:50.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T21:51:50.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=6 2026-02-22T21:51:50.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T21:51:50.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2026-02-22T21:51:50.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2026-02-22T21:51:50.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T21:51:50.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=6 2026-02-22T21:51:50.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2026-02-22T21:51:50.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T21:51:50.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2026-02-22T21:51:50.282 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:51:50.282 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:51:50.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:51:50.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:51:50.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:51:51.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:51:51.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2026-02-22T21:51:51.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2026-02-22T21:51:51.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T21:51:51.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=6 2026-02-22T21:51:51.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T21:51:51.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2026-02-22T21:51:51.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2026-02-22T21:51:52.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2026-02-22T21:51:52.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2026-02-22T21:51:52.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T21:51:52.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=6 2026-02-22T21:51:52.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T21:51:52.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/6 2026-02-22T21:51:52.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:51:52.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:51:52.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:51:52.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:51:52.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2026-02-22T21:51:52.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2026-02-22T21:51:52.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:51:52.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T21:51:52.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:51:52.456 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T21:51:52.456 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:51:52.456 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:51:52.456 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:51:52.456 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:51:52.456 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:51:52.456 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:51:52.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:51:52.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:51:52.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:51:52.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:51:52.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:51:52.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:51:52.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:51:52.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:51:52.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T21:51:52.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T21:51:52.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/6 2026-02-22T21:51:52.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.6 2026-02-22T21:51:52.458 INFO:tasks.workunit.client.0.trial034.stderr:start osd.6 2026-02-22T21:51:52.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 6 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:51:52.458 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/6/whoami 2026-02-22T21:51:52.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 6 = 6 ']' 2026-02-22T21:51:52.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T21:51:52.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T21:51:52.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T21:51:52.474 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:52.470+0000 7ff7d59808c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:51:52.476 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:52.474+0000 7ff7d59808c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:51:52.477 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:52.474+0000 7ff7d59808c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:51:52.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 6 2026-02-22T21:51:52.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:51:52.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2026-02-22T21:51:52.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:51:52.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:51:52.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:52.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:51:52.659 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:51:52.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:52.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:51:52.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:53.789 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:53.786+0000 7ff7d59808c0 -1 Falling back to public interface 2026-02-22T21:51:53.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:53.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:53.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:51:53.855 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:51:53.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:53.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:51:54.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:55.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:55.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:55.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:51:55.048 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:51:55.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:55.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:51:55.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:56.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:56.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:56.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:51:56.242 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:51:56.243 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:56.243 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:51:56.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:51:56.880 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:51:56.878+0000 7ff7d59808c0 -1 osd.6 86 log_to_monitors true 2026-02-22T21:51:57.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:51:57.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:51:57.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:51:57.440 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:51:57.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:51:57.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:51:57.632 INFO:tasks.workunit.client.0.trial034.stderr:osd.6 up in weight 1 up_from 90 up_thru 90 down_at 87 last_clean_interval [30,86) [v2:127.0.0.1:6850/3438080938,v1:127.0.0.1:6851/3438080938] [v2:127.0.0.1:6852/3438080938,v1:127.0.0.1:6853/3438080938] exists,up e712364e-e3c5-4f70-8962-81b340e84b13 2026-02-22T21:51:57.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:51:57.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:51:57.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:51:57.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T21:51:57.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:51:57.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:51:57.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:51:57.633 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:51:57.633 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:51:57.633 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:51:57.633 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:51:57.633 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:51:57.633 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:51:57.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:51:57.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:51:57.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:51:57.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:51:57.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:51:57.736 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:51:57.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:51:57.930 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:51:57.930 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:51:57.930 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:51:57.931 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:51:57.931 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:51:57.931 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:51:57.931 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:51:57.931 INFO:tasks.workunit.client.0.trial034.stderr:8 2026-02-22T21:51:57.931 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:51:57.931 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:51:57.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:51:57.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:57.932 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:51:58.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836518 2026-02-22T21:51:58.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836518 2026-02-22T21:51:58.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836518' 2026-02-22T21:51:58.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:58.007 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:51:58.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949672998 2026-02-22T21:51:58.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949672998 2026-02-22T21:51:58.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836518 1-42949672998' 2026-02-22T21:51:58.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:58.081 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:51:58.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=60129542179 2026-02-22T21:51:58.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 60129542179 2026-02-22T21:51:58.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836518 1-42949672998 2-60129542179' 2026-02-22T21:51:58.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:58.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:51:58.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=292057776140 2026-02-22T21:51:58.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 292057776140 2026-02-22T21:51:58.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836518 1-42949672998 2-60129542179 3-292057776140' 2026-02-22T21:51:58.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:58.231 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:51:58.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=94489280545 2026-02-22T21:51:58.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 94489280545 2026-02-22T21:51:58.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836518 1-42949672998 2-60129542179 3-292057776140 4-94489280545' 2026-02-22T21:51:58.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:58.306 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:51:58.383 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=322122547208 2026-02-22T21:51:58.383 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 322122547208 2026-02-22T21:51:58.383 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836518 1-42949672998 2-60129542179 3-292057776140 4-94489280545 5-322122547208' 2026-02-22T21:51:58.383 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:58.383 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:51:58.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=386547056643 2026-02-22T21:51:58.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 386547056643 2026-02-22T21:51:58.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836518 1-42949672998 2-60129542179 3-292057776140 4-94489280545 5-322122547208 6-386547056643' 2026-02-22T21:51:58.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:58.457 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T21:51:58.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=150323855386 2026-02-22T21:51:58.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 150323855386 2026-02-22T21:51:58.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836518 1-42949672998 2-60129542179 3-292057776140 4-94489280545 5-322122547208 6-386547056643 7-150323855386' 2026-02-22T21:51:58.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:58.533 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T21:51:58.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=171798691864 2026-02-22T21:51:58.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 171798691864 2026-02-22T21:51:58.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836518 1-42949672998 2-60129542179 3-292057776140 4-94489280545 5-322122547208 6-386547056643 7-150323855386 8-171798691864' 2026-02-22T21:51:58.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:58.607 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T21:51:58.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=193273528342 2026-02-22T21:51:58.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 193273528342 2026-02-22T21:51:58.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836518 1-42949672998 2-60129542179 3-292057776140 4-94489280545 5-322122547208 6-386547056643 7-150323855386 8-171798691864 9-193273528342' 2026-02-22T21:51:58.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:51:58.682 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2026-02-22T21:51:58.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=214748364821 2026-02-22T21:51:58.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 214748364821 2026-02-22T21:51:58.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836518 1-42949672998 2-60129542179 3-292057776140 4-94489280545 5-322122547208 6-386547056643 7-150323855386 8-171798691864 9-193273528342 10-214748364821' 2026-02-22T21:51:58.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:51:58.756 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836518 2026-02-22T21:51:58.756 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:51:58.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:51:58.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836518 2026-02-22T21:51:58.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:51:58.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836518 2026-02-22T21:51:58.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2026-02-22T21:51:58.759 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 21474836518 2026-02-22T21:51:58.760 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:51:58.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836516 -lt 21474836518 2026-02-22T21:51:58.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:51:59.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:51:59.958 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:52:00.153 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836519 -lt 21474836518 2026-02-22T21:52:00.153 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:00.154 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949672998 2026-02-22T21:52:00.154 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:00.155 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:52:00.155 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949672998 2026-02-22T21:52:00.156 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:00.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949672998 2026-02-22T21:52:00.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2026-02-22T21:52:00.157 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 42949672998 2026-02-22T21:52:00.158 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:52:00.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949672998 -lt 42949672998 2026-02-22T21:52:00.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:00.361 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-60129542179 2026-02-22T21:52:00.361 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:00.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:52:00.362 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-60129542179 2026-02-22T21:52:00.362 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:00.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=60129542179 2026-02-22T21:52:00.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 60129542179' 2026-02-22T21:52:00.364 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 60129542179 2026-02-22T21:52:00.364 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:52:00.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 60129542179 -lt 60129542179 2026-02-22T21:52:00.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:00.573 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-292057776140 2026-02-22T21:52:00.573 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:00.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:52:00.574 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-292057776140 2026-02-22T21:52:00.575 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:00.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=292057776140 2026-02-22T21:52:00.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 292057776140' 2026-02-22T21:52:00.576 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 292057776140 2026-02-22T21:52:00.576 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:52:00.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 292057776141 -lt 292057776140 2026-02-22T21:52:00.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:00.775 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-94489280545 2026-02-22T21:52:00.776 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:00.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:52:00.776 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-94489280545 2026-02-22T21:52:00.777 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:00.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=94489280545 2026-02-22T21:52:00.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 94489280545' 2026-02-22T21:52:00.778 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.4 seq 94489280545 2026-02-22T21:52:00.779 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:52:00.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 94489280545 -lt 94489280545 2026-02-22T21:52:00.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:00.984 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-322122547208 2026-02-22T21:52:00.984 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:00.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:52:00.986 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-322122547208 2026-02-22T21:52:00.986 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:00.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=322122547208 2026-02-22T21:52:00.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 322122547208' 2026-02-22T21:52:00.987 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.5 seq 322122547208 2026-02-22T21:52:00.988 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:52:01.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 322122547208 -lt 322122547208 2026-02-22T21:52:01.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:01.190 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-386547056643 2026-02-22T21:52:01.191 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:01.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:52:01.192 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-386547056643 2026-02-22T21:52:01.192 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:01.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=386547056643 2026-02-22T21:52:01.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 386547056643' 2026-02-22T21:52:01.194 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.6 seq 386547056643 2026-02-22T21:52:01.194 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:52:01.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 386547056643 -lt 386547056643 2026-02-22T21:52:01.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:01.395 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 7-150323855386 2026-02-22T21:52:01.396 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:01.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=7 2026-02-22T21:52:01.397 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 7-150323855386 2026-02-22T21:52:01.397 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:01.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=150323855386 2026-02-22T21:52:01.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.7 seq 150323855386' 2026-02-22T21:52:01.399 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.7 seq 150323855386 2026-02-22T21:52:01.399 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T21:52:01.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 150323855386 -lt 150323855386 2026-02-22T21:52:01.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:01.600 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 8-171798691864 2026-02-22T21:52:01.601 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:01.601 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=8 2026-02-22T21:52:01.602 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 8-171798691864 2026-02-22T21:52:01.602 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:01.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=171798691864 2026-02-22T21:52:01.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.8 seq 171798691864' 2026-02-22T21:52:01.604 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.8 seq 171798691864 2026-02-22T21:52:01.604 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T21:52:01.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 171798691864 -lt 171798691864 2026-02-22T21:52:01.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:01.798 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 9-193273528342 2026-02-22T21:52:01.798 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:01.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=9 2026-02-22T21:52:01.800 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 9-193273528342 2026-02-22T21:52:01.800 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:01.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=193273528342 2026-02-22T21:52:01.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.9 seq 193273528342' 2026-02-22T21:52:01.801 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.9 seq 193273528342 2026-02-22T21:52:01.801 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T21:52:02.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 193273528343 -lt 193273528342 2026-02-22T21:52:02.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:02.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 10-214748364821 2026-02-22T21:52:02.012 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:02.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=10 2026-02-22T21:52:02.013 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 10-214748364821 2026-02-22T21:52:02.013 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:02.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=214748364821 2026-02-22T21:52:02.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.10 seq 214748364821' 2026-02-22T21:52:02.015 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.10 seq 214748364821 2026-02-22T21:52:02.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-22T21:52:02.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 214748364821 -lt 214748364821 2026-02-22T21:52:02.221 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:52:02.221 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:52:02.222 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:52:02.476 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 == 0 2026-02-22T21:52:02.476 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:52:02.476 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:52:02.477 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:52:02.477 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:52:02.477 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:52:02.477 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:52:02.477 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:52:02.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=28 2026-02-22T21:52:02.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:52:02.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:52:02.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:52:02.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 28 = 28 2026-02-22T21:52:02.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:52:02.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:52:02.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2026-02-22T21:52:02.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2026-02-22T21:52:02.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T21:52:02.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2026-02-22T21:52:02.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T21:52:02.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2026-02-22T21:52:02.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2026-02-22T21:52:02.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T21:52:02.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2026-02-22T21:52:02.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T21:52:02.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2026-02-22T21:52:02.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDremap-pool 2026-02-22T21:52:02.923 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:52:02.923 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:52:02.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:52:02.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:52:02.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:52:04.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:52:04.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2026-02-22T21:52:04.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2026-02-22T21:52:04.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T21:52:04.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2026-02-22T21:52:04.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T21:52:04.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2026-02-22T21:52:04.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2026-02-22T21:52:05.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2026-02-22T21:52:05.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2026-02-22T21:52:05.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T21:52:05.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/9 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:52:05.100 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/9 2026-02-22T21:52:05.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2026-02-22T21:52:05.102 INFO:tasks.workunit.client.0.trial034.stderr:start osd.9 2026-02-22T21:52:05.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=3a8340db-6d94-4027-b5ee-342662db5a1f --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.42545/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:52:05.102 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/9/whoami 2026-02-22T21:52:05.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2026-02-22T21:52:05.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T21:52:05.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T21:52:05.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T21:52:05.119 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:52:05.118+0000 7f61eeace8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:52:05.121 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:52:05.118+0000 7f61eeace8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:52:05.122 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:52:05.118+0000 7f61eeace8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:52:05.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2026-02-22T21:52:05.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:52:05.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2026-02-22T21:52:05.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:52:05.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:52:05.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:52:05.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:52:05.303 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:52:05.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:52:05.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2026-02-22T21:52:05.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:52:06.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:52:06.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:52:06.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:52:06.504 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:52:06.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:52:06.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2026-02-22T21:52:06.624 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:52:06.622+0000 7f61eeace8c0 -1 Falling back to public interface 2026-02-22T21:52:06.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:52:07.703 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:52:07.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:52:07.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:52:07.704 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:52:07.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:52:07.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2026-02-22T21:52:07.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:52:08.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:52:08.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:52:08.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:52:08.899 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:52:08.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:52:08.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2026-02-22T21:52:09.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:52:09.609 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:52:09.606+0000 7f61eeace8c0 -1 osd.9 91 log_to_monitors true 2026-02-22T21:52:10.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:52:10.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:52:10.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:52:10.098 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:52:10.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:52:10.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2026-02-22T21:52:10.293 INFO:tasks.workunit.client.0.trial034.stderr:osd.9 up in weight 1 up_from 95 up_thru 95 down_at 92 last_clean_interval [45,91) [v2:127.0.0.1:6874/1775574881,v1:127.0.0.1:6875/1775574881] [v2:127.0.0.1:6876/1775574881,v1:127.0.0.1:6877/1775574881] exists,up 4f02bd10-132d-4709-9fc7-eca87e892c93 2026-02-22T21:52:10.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:52:10.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:52:10.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:52:10.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T21:52:10.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:52:10.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:52:10.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:52:10.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:52:10.294 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:52:10.294 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:52:10.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:52:10.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:52:10.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:52:10.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:52:10.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:52:10.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:52:10.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:52:10.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:52:10.398 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:52:10.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:52:10.591 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:52:10.592 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:52:10.592 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:52:10.592 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:52:10.592 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:52:10.592 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:52:10.592 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:52:10.592 INFO:tasks.workunit.client.0.trial034.stderr:8 2026-02-22T21:52:10.592 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:52:10.592 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:52:10.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:52:10.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:10.593 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:52:10.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836523 2026-02-22T21:52:10.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836523 2026-02-22T21:52:10.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836523' 2026-02-22T21:52:10.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:10.665 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:52:10.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949673002 2026-02-22T21:52:10.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949673002 2026-02-22T21:52:10.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836523 1-42949673002' 2026-02-22T21:52:10.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:10.740 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:52:10.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=60129542184 2026-02-22T21:52:10.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 60129542184 2026-02-22T21:52:10.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-60129542184' 2026-02-22T21:52:10.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:10.817 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:52:10.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=292057776145 2026-02-22T21:52:10.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 292057776145 2026-02-22T21:52:10.890 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-60129542184 3-292057776145' 2026-02-22T21:52:10.890 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:10.890 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:52:10.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=94489280549 2026-02-22T21:52:10.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 94489280549 2026-02-22T21:52:10.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-60129542184 3-292057776145 4-94489280549' 2026-02-22T21:52:10.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:10.964 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:52:11.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=322122547212 2026-02-22T21:52:11.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 322122547212 2026-02-22T21:52:11.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-60129542184 3-292057776145 4-94489280549 5-322122547212' 2026-02-22T21:52:11.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:11.041 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:52:11.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=386547056647 2026-02-22T21:52:11.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 386547056647 2026-02-22T21:52:11.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-60129542184 3-292057776145 4-94489280549 5-322122547212 6-386547056647' 2026-02-22T21:52:11.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:11.115 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T21:52:11.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=150323855391 2026-02-22T21:52:11.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 150323855391 2026-02-22T21:52:11.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-60129542184 3-292057776145 4-94489280549 5-322122547212 6-386547056647 7-150323855391' 2026-02-22T21:52:11.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:11.195 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T21:52:11.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=171798691869 2026-02-22T21:52:11.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 171798691869 2026-02-22T21:52:11.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-60129542184 3-292057776145 4-94489280549 5-322122547212 6-386547056647 7-150323855391 8-171798691869' 2026-02-22T21:52:11.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:11.269 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T21:52:11.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=408021893123 2026-02-22T21:52:11.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 408021893123 2026-02-22T21:52:11.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-60129542184 3-292057776145 4-94489280549 5-322122547212 6-386547056647 7-150323855391 8-171798691869 9-408021893123' 2026-02-22T21:52:11.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:11.345 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2026-02-22T21:52:11.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=214748364825 2026-02-22T21:52:11.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 214748364825 2026-02-22T21:52:11.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836523 1-42949673002 2-60129542184 3-292057776145 4-94489280549 5-322122547212 6-386547056647 7-150323855391 8-171798691869 9-408021893123 10-214748364825' 2026-02-22T21:52:11.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:11.421 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836523 2026-02-22T21:52:11.421 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:11.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:52:11.422 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836523 2026-02-22T21:52:11.422 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:11.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836523 2026-02-22T21:52:11.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2026-02-22T21:52:11.424 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 21474836523 2026-02-22T21:52:11.424 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:52:11.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836521 -lt 21474836523 2026-02-22T21:52:11.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:52:12.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:52:12.619 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:52:12.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836523 -lt 21474836523 2026-02-22T21:52:12.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:12.826 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949673002 2026-02-22T21:52:12.826 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:12.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:52:12.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949673002 2026-02-22T21:52:12.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:12.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949673002 2026-02-22T21:52:12.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2026-02-22T21:52:12.830 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 42949673002 2026-02-22T21:52:12.830 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:52:13.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949673002 -lt 42949673002 2026-02-22T21:52:13.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:13.023 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-60129542184 2026-02-22T21:52:13.023 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:13.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:52:13.024 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-60129542184 2026-02-22T21:52:13.025 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:13.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=60129542184 2026-02-22T21:52:13.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 60129542184' 2026-02-22T21:52:13.027 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 60129542184 2026-02-22T21:52:13.027 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:52:13.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 60129542184 -lt 60129542184 2026-02-22T21:52:13.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:13.227 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-292057776145 2026-02-22T21:52:13.227 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:13.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:52:13.229 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-292057776145 2026-02-22T21:52:13.229 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:13.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=292057776145 2026-02-22T21:52:13.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 292057776145' 2026-02-22T21:52:13.230 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 292057776145 2026-02-22T21:52:13.231 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:52:13.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 292057776145 -lt 292057776145 2026-02-22T21:52:13.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:13.437 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-94489280549 2026-02-22T21:52:13.437 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:13.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:52:13.439 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-94489280549 2026-02-22T21:52:13.439 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:13.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=94489280549 2026-02-22T21:52:13.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 94489280549' 2026-02-22T21:52:13.440 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.4 seq 94489280549 2026-02-22T21:52:13.440 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:52:13.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 94489280549 -lt 94489280549 2026-02-22T21:52:13.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:13.649 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-322122547212 2026-02-22T21:52:13.649 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:13.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:52:13.651 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-322122547212 2026-02-22T21:52:13.651 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:13.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=322122547212 2026-02-22T21:52:13.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 322122547212' 2026-02-22T21:52:13.652 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.5 seq 322122547212 2026-02-22T21:52:13.652 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:52:13.859 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 322122547212 -lt 322122547212 2026-02-22T21:52:13.859 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:13.860 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-386547056647 2026-02-22T21:52:13.860 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:13.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:52:13.862 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-386547056647 2026-02-22T21:52:13.862 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:13.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=386547056647 2026-02-22T21:52:13.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 386547056647' 2026-02-22T21:52:13.863 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.6 seq 386547056647 2026-02-22T21:52:13.864 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:52:14.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 386547056648 -lt 386547056647 2026-02-22T21:52:14.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:14.072 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 7-150323855391 2026-02-22T21:52:14.072 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:14.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=7 2026-02-22T21:52:14.072 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 7-150323855391 2026-02-22T21:52:14.073 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:14.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=150323855391 2026-02-22T21:52:14.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.7 seq 150323855391' 2026-02-22T21:52:14.074 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.7 seq 150323855391 2026-02-22T21:52:14.074 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T21:52:14.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 150323855391 -lt 150323855391 2026-02-22T21:52:14.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:14.282 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 8-171798691869 2026-02-22T21:52:14.282 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:14.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=8 2026-02-22T21:52:14.283 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 8-171798691869 2026-02-22T21:52:14.283 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:14.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=171798691869 2026-02-22T21:52:14.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.8 seq 171798691869' 2026-02-22T21:52:14.285 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.8 seq 171798691869 2026-02-22T21:52:14.285 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T21:52:14.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 171798691869 -lt 171798691869 2026-02-22T21:52:14.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:14.490 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 9-408021893123 2026-02-22T21:52:14.491 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:14.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=9 2026-02-22T21:52:14.492 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 9-408021893123 2026-02-22T21:52:14.492 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:14.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=408021893123 2026-02-22T21:52:14.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.9 seq 408021893123' 2026-02-22T21:52:14.494 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.9 seq 408021893123 2026-02-22T21:52:14.494 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T21:52:14.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 408021893123 -lt 408021893123 2026-02-22T21:52:14.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:14.701 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 10-214748364825 2026-02-22T21:52:14.701 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:14.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=10 2026-02-22T21:52:14.703 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 10-214748364825 2026-02-22T21:52:14.703 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:14.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=214748364825 2026-02-22T21:52:14.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.10 seq 214748364825' 2026-02-22T21:52:14.704 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.10 seq 214748364825 2026-02-22T21:52:14.704 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-22T21:52:14.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 214748364826 -lt 214748364825 2026-02-22T21:52:14.898 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:52:14.899 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:52:14.899 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:52:15.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 == 0 2026-02-22T21:52:15.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:52:15.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:52:15.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:52:15.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:52:15.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:52:15.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:52:15.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:52:15.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=28 2026-02-22T21:52:15.350 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:52:15.350 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:52:15.350 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:52:15.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 28 = 28 2026-02-22T21:52:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:52:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:52:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:329: TEST_chunk_mapping: delete_pool remap-pool 2026-02-22T21:52:15.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=remap-pool 2026-02-22T21:52:15.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2026-02-22T21:52:15.819 INFO:tasks.workunit.client.0.trial034.stderr:pool 'remap-pool' does not exist 2026-02-22T21:52:15.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:330: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2026-02-22T21:52:16.123 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile remap-profile does not exist 2026-02-22T21:52:16.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-22T21:52:16.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2026-02-22T21:52:16.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2026-02-22T21:52:16.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: erasure_code_plugin_exists: local plugin=isa 2026-02-22T21:52:16.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: erasure_code_plugin_exists: local status 2026-02-22T21:52:16.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: erasure_code_plugin_exists: local grepstr 2026-02-22T21:52:16.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: erasure_code_plugin_exists: local s 2026-02-22T21:52:16.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2118: erasure_code_plugin_exists: case `uname` in 2026-02-22T21:52:16.135 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2118: erasure_code_plugin_exists: uname 2026-02-22T21:52:16.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: erasure_code_plugin_exists: grepstr='isa.*No such file' 2026-02-22T21:52:16.135 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2026-02-22T21:52:16.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: erasure_code_plugin_exists: s= 2026-02-22T21:52:16.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: erasure_code_plugin_exists: local status=0 2026-02-22T21:52:16.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2026-02-22T21:52:16.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2026-02-22T21:52:16.638 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile TESTPROFILE does not exist 2026-02-22T21:52:16.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: erasure_code_plugin_exists: return 0 2026-02-22T21:52:16.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:197: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2026-02-22T21:52:16.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:198: TEST_rados_put_get_isa: local poolname=pool-isa 2026-02-22T21:52:16.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:200: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2026-02-22T21:52:16.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:203: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2026-02-22T21:52:16.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2026-02-22T21:52:17.273 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-isa' already exists 2026-02-22T21:52:17.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:52:18.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:206: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2026-02-22T21:52:18.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-22T21:52:18.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2026-02-22T21:52:18.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-22T21:52:18.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:18.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-22T21:52:18.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:18.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-22T21:52:18.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:18.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-22T21:52:18.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:18.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-22T21:52:18.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-22T21:52:18.310 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:52:18.329 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:52:18.330 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-22T21:52:18.331 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2026-02-22T21:52:18.331 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2026-02-22T21:52:18.332 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:52:18.332 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2026-02-22T21:52:18.332 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:52:18.529 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2026-02-22T21:52:18.529 INFO:tasks.workunit.client.0.trial034.stderr:10 2026-02-22T21:52:18.529 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:52:18.529 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:52:18.529 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:52:18.529 INFO:tasks.workunit.client.0.trial034.stderr:8 2026-02-22T21:52:18.529 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:52:18.529 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:52:18.529 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:52:18.530 INFO:tasks.workunit.client.0.trial034.stderr:9' 2026-02-22T21:52:18.530 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 2 1 5 9 2026-02-22T21:52:18.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('4' '10' '0' '7' '6' '8' '2' '1' '5' '9') 2026-02-22T21:52:18.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-22T21:52:18.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=9 2026-02-22T21:52:18.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2026-02-22T21:52:18.776 INFO:tasks.workunit.client.0.trial034.stderr:osd.9 is already out. 2026-02-22T21:52:18.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-22T21:52:23.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-isa SOMETHING 2026-02-22T21:52:23.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2026-02-22T21:52:23.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:52:23.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2026-02-22T21:52:23.788 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2026-02-22T21:52:23.788 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:52:23.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2026-02-22T21:52:23.988 INFO:tasks.workunit.client.0.trial034.stderr:10 2026-02-22T21:52:23.989 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:52:23.989 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:52:23.989 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:52:23.989 INFO:tasks.workunit.client.0.trial034.stderr:8 2026-02-22T21:52:23.989 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:52:23.989 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:52:23.989 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:52:23.989 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T21:52:23.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 3 1 5 2 2026-02-22T21:52:23.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:52:24.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:52:24.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2026-02-22T21:52:24.215 INFO:tasks.workunit.client.0.trial034.stderr:osd.9 is already in. 2026-02-22T21:52:24.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-22T21:52:24.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_isa: delete_pool pool-isa 2026-02-22T21:52:24.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-isa 2026-02-22T21:52:24.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2026-02-22T21:52:24.516 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-isa' does not exist 2026-02-22T21:52:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-22T21:52:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2026-02-22T21:52:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2026-02-22T21:52:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:214: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2026-02-22T21:52:24.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-22T21:52:24.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2026-02-22T21:52:24.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-22T21:52:24.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:24.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-22T21:52:24.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:24.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-22T21:52:24.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:24.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-22T21:52:24.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:24.529 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-22T21:52:24.529 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-22T21:52:24.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:52:24.578 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:52:24.579 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-22T21:52:24.581 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2026-02-22T21:52:24.581 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2026-02-22T21:52:24.581 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:52:24.582 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-22T21:52:24.582 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:52:24.775 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2026-02-22T21:52:24.775 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:52:24.775 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:52:24.775 INFO:tasks.workunit.client.0.trial034.stderr:9' 2026-02-22T21:52:24.775 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 9 2026-02-22T21:52:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('6' '0' '7' '9') 2026-02-22T21:52:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-22T21:52:24.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2026-02-22T21:52:24.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2026-02-22T21:52:25.025 INFO:tasks.workunit.client.0.trial034.stderr:osd.9 is already out. 2026-02-22T21:52:25.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-22T21:52:30.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds ecpool SOMETHING 2026-02-22T21:52:30.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2026-02-22T21:52:30.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:52:30.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2026-02-22T21:52:30.038 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-22T21:52:30.038 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:52:30.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2026-02-22T21:52:30.241 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:52:30.241 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:52:30.241 INFO:tasks.workunit.client.0.trial034.stderr:5' 2026-02-22T21:52:30.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 5 2026-02-22T21:52:30.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:52:30.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:52:30.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2026-02-22T21:52:30.511 INFO:tasks.workunit.client.0.trial034.stderr:osd.9 is already in. 2026-02-22T21:52:30.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-22T21:52:30.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:216: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2026-02-22T21:52:30.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:217: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2026-02-22T21:52:30.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2026-02-22T21:52:30.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2026-02-22T21:52:30.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2026-02-22T21:52:31.167 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T21:52:31.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:52:32.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2026-02-22T21:52:32.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-22T21:52:32.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2026-02-22T21:52:32.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-22T21:52:32.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:32.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-22T21:52:32.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:32.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-22T21:52:32.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:32.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-22T21:52:32.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:32.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-22T21:52:32.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-22T21:52:32.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:52:32.218 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:52:32.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-22T21:52:32.221 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2026-02-22T21:52:32.221 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:52:32.221 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:52:32.221 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-22T21:52:32.221 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:52:32.429 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2026-02-22T21:52:32.429 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:52:32.429 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:52:32.429 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:52:32.429 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:52:32.429 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:52:32.429 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2026-02-22T21:52:32.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '1' '9' '4' '3' '10') 2026-02-22T21:52:32.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-22T21:52:32.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2026-02-22T21:52:32.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2026-02-22T21:52:32.671 INFO:tasks.workunit.client.0.trial034.stderr:osd.10 is already out. 2026-02-22T21:52:32.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-22T21:52:37.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-jerasure SOMETHING 2026-02-22T21:52:37.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:52:37.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:52:37.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<10\>' 2026-02-22T21:52:37.684 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-22T21:52:37.684 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:52:37.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2026-02-22T21:52:37.888 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:52:37.889 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:52:37.889 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:52:37.889 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:52:37.889 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T21:52:37.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 6 2026-02-22T21:52:37.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:52:37.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:52:37.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 10 2026-02-22T21:52:38.157 INFO:tasks.workunit.client.0.trial034.stderr:osd.10 is already in. 2026-02-22T21:52:38.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-22T21:52:38.168 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:227: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2026-02-22T21:52:38.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:104: rados_osds_out_in: local dir=td/test-erasure-code 2026-02-22T21:52:38.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: local poolname=pool-jerasure 2026-02-22T21:52:38.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: local objname=SOMETHING 2026-02-22T21:52:38.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2026-02-22T21:52:38.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 FFFF 2026-02-22T21:52:38.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2026-02-22T21:52:38.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 GGGG 2026-02-22T21:52:38.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2026-02-22T21:52:38.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 HHHH 2026-02-22T21:52:38.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2026-02-22T21:52:38.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 IIII 2026-02-22T21:52:38.170 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:116: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-22T21:52:38.646 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:117: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:52:38.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:118: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:52:38.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:119: rados_osds_out_in: rm td/test-erasure-code/COPY 2026-02-22T21:52:38.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: wait_for_clean 2026-02-22T21:52:38.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:52:38.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:52:38.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:52:38.668 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:52:38.668 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:52:38.668 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:52:38.668 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:52:38.668 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:52:38.668 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:52:38.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:52:38.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:52:38.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:52:38.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:52:38.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:52:38.769 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:52:38.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:52:38.962 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:52:38.962 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:52:38.963 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:52:38.963 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:52:38.963 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:52:38.963 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:52:38.963 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:52:38.963 INFO:tasks.workunit.client.0.trial034.stderr:8 2026-02-22T21:52:38.963 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:52:38.963 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:52:38.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:52:38.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:38.964 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:52:39.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836531 2026-02-22T21:52:39.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836531 2026-02-22T21:52:39.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836531' 2026-02-22T21:52:39.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:39.038 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:52:39.113 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949673010 2026-02-22T21:52:39.113 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949673010 2026-02-22T21:52:39.113 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836531 1-42949673010' 2026-02-22T21:52:39.113 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:39.114 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:52:39.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=60129542191 2026-02-22T21:52:39.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 60129542191 2026-02-22T21:52:39.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836531 1-42949673010 2-60129542191' 2026-02-22T21:52:39.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:39.188 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:52:39.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=292057776153 2026-02-22T21:52:39.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 292057776153 2026-02-22T21:52:39.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836531 1-42949673010 2-60129542191 3-292057776153' 2026-02-22T21:52:39.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:39.265 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:52:39.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=94489280557 2026-02-22T21:52:39.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 94489280557 2026-02-22T21:52:39.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836531 1-42949673010 2-60129542191 3-292057776153 4-94489280557' 2026-02-22T21:52:39.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:39.337 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:52:39.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=322122547220 2026-02-22T21:52:39.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 322122547220 2026-02-22T21:52:39.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836531 1-42949673010 2-60129542191 3-292057776153 4-94489280557 5-322122547220' 2026-02-22T21:52:39.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:39.413 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:52:39.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=386547056655 2026-02-22T21:52:39.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 386547056655 2026-02-22T21:52:39.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836531 1-42949673010 2-60129542191 3-292057776153 4-94489280557 5-322122547220 6-386547056655' 2026-02-22T21:52:39.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:39.488 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T21:52:39.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=150323855398 2026-02-22T21:52:39.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 150323855398 2026-02-22T21:52:39.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836531 1-42949673010 2-60129542191 3-292057776153 4-94489280557 5-322122547220 6-386547056655 7-150323855398' 2026-02-22T21:52:39.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:39.565 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T21:52:39.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=171798691876 2026-02-22T21:52:39.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 171798691876 2026-02-22T21:52:39.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836531 1-42949673010 2-60129542191 3-292057776153 4-94489280557 5-322122547220 6-386547056655 7-150323855398 8-171798691876' 2026-02-22T21:52:39.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:39.640 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T21:52:39.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=408021893131 2026-02-22T21:52:39.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 408021893131 2026-02-22T21:52:39.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836531 1-42949673010 2-60129542191 3-292057776153 4-94489280557 5-322122547220 6-386547056655 7-150323855398 8-171798691876 9-408021893131' 2026-02-22T21:52:39.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:39.716 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2026-02-22T21:52:39.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=214748364833 2026-02-22T21:52:39.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 214748364833 2026-02-22T21:52:39.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836531 1-42949673010 2-60129542191 3-292057776153 4-94489280557 5-322122547220 6-386547056655 7-150323855398 8-171798691876 9-408021893131 10-214748364833' 2026-02-22T21:52:39.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:39.790 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836531 2026-02-22T21:52:39.791 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:39.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:52:39.792 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836531 2026-02-22T21:52:39.792 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:39.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836531 2026-02-22T21:52:39.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2026-02-22T21:52:39.794 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 21474836531 2026-02-22T21:52:39.794 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:52:39.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836531 -lt 21474836531 2026-02-22T21:52:39.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:39.998 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949673010 2026-02-22T21:52:39.998 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:39.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:52:40.000 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949673010 2026-02-22T21:52:40.001 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:40.002 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949673010 2026-02-22T21:52:40.002 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 42949673010 2026-02-22T21:52:40.002 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2026-02-22T21:52:40.002 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:52:40.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949673010 -lt 42949673010 2026-02-22T21:52:40.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:40.207 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-60129542191 2026-02-22T21:52:40.207 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:52:40.209 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-60129542191 2026-02-22T21:52:40.209 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:40.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=60129542191 2026-02-22T21:52:40.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 60129542191' 2026-02-22T21:52:40.211 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 60129542191 2026-02-22T21:52:40.211 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:52:40.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 60129542191 -lt 60129542191 2026-02-22T21:52:40.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:40.419 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-292057776153 2026-02-22T21:52:40.419 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:40.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:52:40.421 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-292057776153 2026-02-22T21:52:40.421 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:40.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=292057776153 2026-02-22T21:52:40.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 292057776153' 2026-02-22T21:52:40.423 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 292057776153 2026-02-22T21:52:40.424 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:52:40.627 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 292057776153 -lt 292057776153 2026-02-22T21:52:40.627 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:40.628 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-94489280557 2026-02-22T21:52:40.628 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:40.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:52:40.630 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-94489280557 2026-02-22T21:52:40.630 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:40.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=94489280557 2026-02-22T21:52:40.631 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 94489280557 2026-02-22T21:52:40.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 94489280557' 2026-02-22T21:52:40.632 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:52:40.834 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 94489280557 -lt 94489280557 2026-02-22T21:52:40.834 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:40.835 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-322122547220 2026-02-22T21:52:40.835 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:40.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:52:40.837 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-322122547220 2026-02-22T21:52:40.837 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:40.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=322122547220 2026-02-22T21:52:40.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 322122547220' 2026-02-22T21:52:40.838 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 322122547220 2026-02-22T21:52:40.839 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:52:41.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 322122547220 -lt 322122547220 2026-02-22T21:52:41.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:41.042 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-386547056655 2026-02-22T21:52:41.042 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:41.043 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:52:41.044 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-386547056655 2026-02-22T21:52:41.044 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:41.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=386547056655 2026-02-22T21:52:41.046 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 386547056655 2026-02-22T21:52:41.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 386547056655' 2026-02-22T21:52:41.046 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:52:41.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 386547056655 -lt 386547056655 2026-02-22T21:52:41.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:41.253 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 7-150323855398 2026-02-22T21:52:41.253 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:41.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=7 2026-02-22T21:52:41.254 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 7-150323855398 2026-02-22T21:52:41.254 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:41.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=150323855398 2026-02-22T21:52:41.256 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.7 seq 150323855398 2026-02-22T21:52:41.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.7 seq 150323855398' 2026-02-22T21:52:41.256 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T21:52:41.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 150323855398 -lt 150323855398 2026-02-22T21:52:41.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:41.457 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 8-171798691876 2026-02-22T21:52:41.457 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:41.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=8 2026-02-22T21:52:41.459 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 8-171798691876 2026-02-22T21:52:41.459 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:41.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=171798691876 2026-02-22T21:52:41.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.8 seq 171798691876' 2026-02-22T21:52:41.461 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.8 seq 171798691876 2026-02-22T21:52:41.461 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T21:52:41.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 171798691876 -lt 171798691876 2026-02-22T21:52:41.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:41.655 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 9-408021893131 2026-02-22T21:52:41.655 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:41.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=9 2026-02-22T21:52:41.658 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 9-408021893131 2026-02-22T21:52:41.658 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:41.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=408021893131 2026-02-22T21:52:41.659 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.9 seq 408021893131 2026-02-22T21:52:41.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.9 seq 408021893131' 2026-02-22T21:52:41.660 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T21:52:41.860 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 408021893131 -lt 408021893131 2026-02-22T21:52:41.860 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:41.861 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 10-214748364833 2026-02-22T21:52:41.861 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:41.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=10 2026-02-22T21:52:41.862 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 10-214748364833 2026-02-22T21:52:41.862 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:41.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=214748364833 2026-02-22T21:52:41.864 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.10 seq 214748364833 2026-02-22T21:52:41.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.10 seq 214748364833' 2026-02-22T21:52:41.864 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-22T21:52:42.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 214748364833 -lt 214748364833 2026-02-22T21:52:42.059 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:52:42.059 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:52:42.060 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:52:42.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 == 0 2026-02-22T21:52:42.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:52:42.317 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:52:42.317 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:52:42.317 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:52:42.317 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:52:42.317 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:52:42.317 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:52:42.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=28 2026-02-22T21:52:42.511 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:52:42.511 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:52:42.511 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:52:42.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 28 = 28 2026-02-22T21:52:42.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:52:42.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:52:42.767 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2026-02-22T21:52:42.767 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:52:42.767 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:52:42.767 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-22T21:52:42.767 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:52:42.970 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2026-02-22T21:52:42.970 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:52:42.970 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:52:42.970 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:52:42.971 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:52:42.971 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:52:42.971 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2026-02-22T21:52:42.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2026-02-22T21:52:42.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: osds=('5' '1' '9' '4' '3' '10') 2026-02-22T21:52:42.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: local -a osds 2026-02-22T21:52:42.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2026-02-22T21:52:42.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 5 2026-02-22T21:52:43.212 INFO:tasks.workunit.client.0.trial034.stderr:osd.5 is already out. 2026-02-22T21:52:43.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2026-02-22T21:52:43.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 1 2026-02-22T21:52:43.516 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 is already out. 2026-02-22T21:52:43.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: wait_for_clean 2026-02-22T21:52:43.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:52:43.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:52:43.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:52:43.528 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:52:43.528 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:52:43.528 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:52:43.529 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:52:43.529 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:52:43.529 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:52:43.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:52:43.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:52:43.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:52:43.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:52:43.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:52:43.634 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:52:43.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:52:43.828 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:52:43.828 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:52:43.828 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:52:43.828 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:52:43.828 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:52:43.829 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:52:43.829 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:52:43.829 INFO:tasks.workunit.client.0.trial034.stderr:8 2026-02-22T21:52:43.829 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:52:43.829 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:52:43.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:52:43.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:43.829 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:52:43.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836534 2026-02-22T21:52:43.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836534 2026-02-22T21:52:43.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836534' 2026-02-22T21:52:43.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:43.906 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:52:43.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949673013 2026-02-22T21:52:43.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949673013 2026-02-22T21:52:43.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836534 1-42949673013' 2026-02-22T21:52:43.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:43.981 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:52:44.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=60129542194 2026-02-22T21:52:44.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 60129542194 2026-02-22T21:52:44.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836534 1-42949673013 2-60129542194' 2026-02-22T21:52:44.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:44.056 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:52:44.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=292057776156 2026-02-22T21:52:44.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 292057776156 2026-02-22T21:52:44.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836534 1-42949673013 2-60129542194 3-292057776156' 2026-02-22T21:52:44.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:44.133 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:52:44.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=94489280560 2026-02-22T21:52:44.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 94489280560 2026-02-22T21:52:44.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836534 1-42949673013 2-60129542194 3-292057776156 4-94489280560' 2026-02-22T21:52:44.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:44.209 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:52:44.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=322122547223 2026-02-22T21:52:44.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 322122547223 2026-02-22T21:52:44.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836534 1-42949673013 2-60129542194 3-292057776156 4-94489280560 5-322122547223' 2026-02-22T21:52:44.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:44.285 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:52:44.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=386547056658 2026-02-22T21:52:44.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 386547056658 2026-02-22T21:52:44.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836534 1-42949673013 2-60129542194 3-292057776156 4-94489280560 5-322122547223 6-386547056658' 2026-02-22T21:52:44.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:44.361 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T21:52:44.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=150323855401 2026-02-22T21:52:44.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 150323855401 2026-02-22T21:52:44.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836534 1-42949673013 2-60129542194 3-292057776156 4-94489280560 5-322122547223 6-386547056658 7-150323855401' 2026-02-22T21:52:44.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:44.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T21:52:44.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=171798691879 2026-02-22T21:52:44.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 171798691879 2026-02-22T21:52:44.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836534 1-42949673013 2-60129542194 3-292057776156 4-94489280560 5-322122547223 6-386547056658 7-150323855401 8-171798691879' 2026-02-22T21:52:44.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:44.512 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T21:52:44.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=408021893133 2026-02-22T21:52:44.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 408021893133 2026-02-22T21:52:44.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836534 1-42949673013 2-60129542194 3-292057776156 4-94489280560 5-322122547223 6-386547056658 7-150323855401 8-171798691879 9-408021893133' 2026-02-22T21:52:44.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:44.589 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2026-02-22T21:52:44.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=214748364836 2026-02-22T21:52:44.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 214748364836 2026-02-22T21:52:44.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836534 1-42949673013 2-60129542194 3-292057776156 4-94489280560 5-322122547223 6-386547056658 7-150323855401 8-171798691879 9-408021893133 10-214748364836' 2026-02-22T21:52:44.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:44.665 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836534 2026-02-22T21:52:44.665 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:44.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:52:44.667 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836534 2026-02-22T21:52:44.667 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:44.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836534 2026-02-22T21:52:44.669 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 21474836534 2026-02-22T21:52:44.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2026-02-22T21:52:44.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:52:44.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836532 -lt 21474836534 2026-02-22T21:52:44.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:52:45.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:52:45.875 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:52:46.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836534 -lt 21474836534 2026-02-22T21:52:46.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:46.067 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949673013 2026-02-22T21:52:46.067 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:46.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:52:46.069 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949673013 2026-02-22T21:52:46.069 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:46.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949673013 2026-02-22T21:52:46.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949673013' 2026-02-22T21:52:46.071 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 42949673013 2026-02-22T21:52:46.071 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:52:46.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949673013 -lt 42949673013 2026-02-22T21:52:46.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:46.275 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-60129542194 2026-02-22T21:52:46.276 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:46.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:52:46.277 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-60129542194 2026-02-22T21:52:46.278 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:46.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=60129542194 2026-02-22T21:52:46.279 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 60129542194 2026-02-22T21:52:46.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 60129542194' 2026-02-22T21:52:46.280 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:52:46.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 60129542195 -lt 60129542194 2026-02-22T21:52:46.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:46.475 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-292057776156 2026-02-22T21:52:46.476 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:46.476 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:52:46.477 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-292057776156 2026-02-22T21:52:46.477 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:46.478 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=292057776156 2026-02-22T21:52:46.479 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 292057776156 2026-02-22T21:52:46.479 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 292057776156' 2026-02-22T21:52:46.479 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:52:46.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 292057776156 -lt 292057776156 2026-02-22T21:52:46.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:46.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-94489280560 2026-02-22T21:52:46.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:46.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:52:46.690 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-94489280560 2026-02-22T21:52:46.690 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:46.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=94489280560 2026-02-22T21:52:46.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 94489280560' 2026-02-22T21:52:46.692 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 94489280560 2026-02-22T21:52:46.692 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:52:46.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 94489280560 -lt 94489280560 2026-02-22T21:52:46.890 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:46.890 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-322122547223 2026-02-22T21:52:46.890 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:46.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:52:46.892 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-322122547223 2026-02-22T21:52:46.893 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:46.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=322122547223 2026-02-22T21:52:46.894 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 322122547223 2026-02-22T21:52:46.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 322122547223' 2026-02-22T21:52:46.895 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:52:47.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 322122547223 -lt 322122547223 2026-02-22T21:52:47.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:47.094 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-386547056658 2026-02-22T21:52:47.094 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:47.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:52:47.095 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-386547056658 2026-02-22T21:52:47.096 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:47.097 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=386547056658 2026-02-22T21:52:47.097 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 386547056658 2026-02-22T21:52:47.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 386547056658' 2026-02-22T21:52:47.098 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:52:47.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 386547056658 -lt 386547056658 2026-02-22T21:52:47.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:47.293 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 7-150323855401 2026-02-22T21:52:47.293 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:47.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=7 2026-02-22T21:52:47.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 7-150323855401 2026-02-22T21:52:47.296 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:47.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=150323855401 2026-02-22T21:52:47.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.7 seq 150323855401' 2026-02-22T21:52:47.297 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.7 seq 150323855401 2026-02-22T21:52:47.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T21:52:47.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 150323855402 -lt 150323855401 2026-02-22T21:52:47.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:47.506 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 8-171798691879 2026-02-22T21:52:47.506 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:47.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=8 2026-02-22T21:52:47.508 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 8-171798691879 2026-02-22T21:52:47.508 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:47.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=171798691879 2026-02-22T21:52:47.510 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.8 seq 171798691879 2026-02-22T21:52:47.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.8 seq 171798691879' 2026-02-22T21:52:47.510 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T21:52:47.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 171798691880 -lt 171798691879 2026-02-22T21:52:47.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:47.717 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 9-408021893133 2026-02-22T21:52:47.717 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:47.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=9 2026-02-22T21:52:47.719 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 9-408021893133 2026-02-22T21:52:47.719 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:47.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=408021893133 2026-02-22T21:52:47.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.9 seq 408021893133' 2026-02-22T21:52:47.721 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.9 seq 408021893133 2026-02-22T21:52:47.721 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T21:52:47.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 408021893134 -lt 408021893133 2026-02-22T21:52:47.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:47.932 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 10-214748364836 2026-02-22T21:52:47.932 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:47.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=10 2026-02-22T21:52:47.934 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 10-214748364836 2026-02-22T21:52:47.934 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:47.936 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=214748364836 2026-02-22T21:52:47.936 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.10 seq 214748364836 2026-02-22T21:52:47.936 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.10 seq 214748364836' 2026-02-22T21:52:47.936 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-22T21:52:48.129 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 214748364836 -lt 214748364836 2026-02-22T21:52:48.130 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:52:48.130 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:52:48.130 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:52:48.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 == 0 2026-02-22T21:52:48.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:52:48.393 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:52:48.393 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:52:48.393 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:52:48.393 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:52:48.393 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:52:48.393 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:52:48.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=28 2026-02-22T21:52:48.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:52:48.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:52:48.585 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:52:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 28 = 28 2026-02-22T21:52:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:52:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:52:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2026-02-22T21:52:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2026-02-22T21:52:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:52:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:52:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<5\>' 2026-02-22T21:52:48.844 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-22T21:52:48.844 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:52:49.044 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2026-02-22T21:52:49.045 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:52:49.045 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:52:49.045 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:52:49.045 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:52:49.045 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:52:49.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2026-02-22T21:52:49.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2026-02-22T21:52:49.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2026-02-22T21:52:49.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:52:49.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:52:49.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<1\>' 2026-02-22T21:52:49.046 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-22T21:52:49.046 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:52:49.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2026-02-22T21:52:49.241 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:52:49.241 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:52:49.241 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:52:49.242 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:52:49.242 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:52:49.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2026-02-22T21:52:49.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:140: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:52:49.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:141: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:52:49.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2026-02-22T21:52:49.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 5 2026-02-22T21:52:49.507 INFO:tasks.workunit.client.0.trial034.stderr:osd.5 is already in. 2026-02-22T21:52:49.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2026-02-22T21:52:49.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 1 2026-02-22T21:52:49.813 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 is already in. 2026-02-22T21:52:49.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:150: rados_osds_out_in: wait_for_clean 2026-02-22T21:52:49.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:52:49.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:52:49.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:52:49.824 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:52:49.824 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:52:49.824 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:52:49.825 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:52:49.825 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:52:49.825 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:52:49.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:52:49.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:52:49.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:52:49.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:52:49.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:52:49.933 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:8 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:52:50.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:50.132 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:52:50.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=21474836537 2026-02-22T21:52:50.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 21474836537 2026-02-22T21:52:50.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836537' 2026-02-22T21:52:50.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:50.211 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:52:50.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=42949673016 2026-02-22T21:52:50.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 42949673016 2026-02-22T21:52:50.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836537 1-42949673016' 2026-02-22T21:52:50.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:50.287 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:52:50.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=60129542197 2026-02-22T21:52:50.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 60129542197 2026-02-22T21:52:50.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836537 1-42949673016 2-60129542197' 2026-02-22T21:52:50.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:50.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:52:50.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=292057776159 2026-02-22T21:52:50.439 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 292057776159 2026-02-22T21:52:50.439 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836537 1-42949673016 2-60129542197 3-292057776159' 2026-02-22T21:52:50.439 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:50.439 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:52:50.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=94489280563 2026-02-22T21:52:50.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 94489280563 2026-02-22T21:52:50.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836537 1-42949673016 2-60129542197 3-292057776159 4-94489280563' 2026-02-22T21:52:50.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:50.514 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:52:50.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=322122547226 2026-02-22T21:52:50.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 322122547226 2026-02-22T21:52:50.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836537 1-42949673016 2-60129542197 3-292057776159 4-94489280563 5-322122547226' 2026-02-22T21:52:50.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:50.591 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:52:50.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=386547056661 2026-02-22T21:52:50.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 386547056661 2026-02-22T21:52:50.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836537 1-42949673016 2-60129542197 3-292057776159 4-94489280563 5-322122547226 6-386547056661' 2026-02-22T21:52:50.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:50.666 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T21:52:50.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=150323855404 2026-02-22T21:52:50.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 150323855404 2026-02-22T21:52:50.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836537 1-42949673016 2-60129542197 3-292057776159 4-94489280563 5-322122547226 6-386547056661 7-150323855404' 2026-02-22T21:52:50.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:50.741 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T21:52:50.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=171798691883 2026-02-22T21:52:50.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 171798691883 2026-02-22T21:52:50.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836537 1-42949673016 2-60129542197 3-292057776159 4-94489280563 5-322122547226 6-386547056661 7-150323855404 8-171798691883' 2026-02-22T21:52:50.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:50.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T21:52:50.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=408021893137 2026-02-22T21:52:50.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 408021893137 2026-02-22T21:52:50.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836537 1-42949673016 2-60129542197 3-292057776159 4-94489280563 5-322122547226 6-386547056661 7-150323855404 8-171798691883 9-408021893137' 2026-02-22T21:52:50.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:52:50.897 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2026-02-22T21:52:50.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=214748364839 2026-02-22T21:52:50.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 214748364839 2026-02-22T21:52:50.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-21474836537 1-42949673016 2-60129542197 3-292057776159 4-94489280563 5-322122547226 6-386547056661 7-150323855404 8-171798691883 9-408021893137 10-214748364839' 2026-02-22T21:52:50.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:50.973 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-21474836537 2026-02-22T21:52:50.974 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:50.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:52:50.975 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-21474836537 2026-02-22T21:52:50.975 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:50.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=21474836537 2026-02-22T21:52:50.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2026-02-22T21:52:50.977 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 21474836537 2026-02-22T21:52:50.977 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:52:51.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836535 -lt 21474836537 2026-02-22T21:52:51.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:52:52.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:52:52.190 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:52:52.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 21474836537 -lt 21474836537 2026-02-22T21:52:52.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:52.394 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-42949673016 2026-02-22T21:52:52.395 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:52.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:52:52.396 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-42949673016 2026-02-22T21:52:52.397 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:52.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=42949673016 2026-02-22T21:52:52.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 42949673016' 2026-02-22T21:52:52.398 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 42949673016 2026-02-22T21:52:52.399 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:52:52.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 42949673016 -lt 42949673016 2026-02-22T21:52:52.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:52.605 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-60129542197 2026-02-22T21:52:52.605 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:52.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:52:52.608 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-60129542197 2026-02-22T21:52:52.608 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:52.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=60129542197 2026-02-22T21:52:52.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 60129542197' 2026-02-22T21:52:52.610 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 60129542197 2026-02-22T21:52:52.610 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:52:52.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 60129542198 -lt 60129542197 2026-02-22T21:52:52.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:52.823 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-292057776159 2026-02-22T21:52:52.824 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:52.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:52:52.826 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-292057776159 2026-02-22T21:52:52.826 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:52.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=292057776159 2026-02-22T21:52:52.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 292057776159' 2026-02-22T21:52:52.828 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 292057776159 2026-02-22T21:52:52.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:52:53.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 292057776159 -lt 292057776159 2026-02-22T21:52:53.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:53.032 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-94489280563 2026-02-22T21:52:53.032 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:53.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:52:53.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-94489280563 2026-02-22T21:52:53.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:53.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=94489280563 2026-02-22T21:52:53.036 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 94489280563 2026-02-22T21:52:53.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 94489280563' 2026-02-22T21:52:53.036 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:52:53.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 94489280563 -lt 94489280563 2026-02-22T21:52:53.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:53.236 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-322122547226 2026-02-22T21:52:53.236 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:53.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:52:53.237 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-322122547226 2026-02-22T21:52:53.238 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:53.239 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=322122547226 2026-02-22T21:52:53.239 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 322122547226' 2026-02-22T21:52:53.239 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 322122547226 2026-02-22T21:52:53.240 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:52:53.432 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 322122547226 -lt 322122547226 2026-02-22T21:52:53.433 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:53.433 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-386547056661 2026-02-22T21:52:53.433 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:53.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:52:53.435 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-386547056661 2026-02-22T21:52:53.435 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:53.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=386547056661 2026-02-22T21:52:53.436 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 386547056661 2026-02-22T21:52:53.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 386547056661' 2026-02-22T21:52:53.437 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:52:53.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 386547056661 -lt 386547056661 2026-02-22T21:52:53.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:53.634 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 7-150323855404 2026-02-22T21:52:53.634 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:53.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=7 2026-02-22T21:52:53.636 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 7-150323855404 2026-02-22T21:52:53.636 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:53.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=150323855404 2026-02-22T21:52:53.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.7 seq 150323855404' 2026-02-22T21:52:53.638 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.7 seq 150323855404 2026-02-22T21:52:53.638 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T21:52:53.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 150323855405 -lt 150323855404 2026-02-22T21:52:53.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:53.849 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 8-171798691883 2026-02-22T21:52:53.849 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:53.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=8 2026-02-22T21:52:53.851 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 8-171798691883 2026-02-22T21:52:53.851 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:53.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=171798691883 2026-02-22T21:52:53.853 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.8 seq 171798691883 2026-02-22T21:52:53.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.8 seq 171798691883' 2026-02-22T21:52:53.853 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T21:52:54.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 171798691883 -lt 171798691883 2026-02-22T21:52:54.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:54.050 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 9-408021893137 2026-02-22T21:52:54.050 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:54.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=9 2026-02-22T21:52:54.052 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 9-408021893137 2026-02-22T21:52:54.052 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:54.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=408021893137 2026-02-22T21:52:54.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.9 seq 408021893137' 2026-02-22T21:52:54.053 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.9 seq 408021893137 2026-02-22T21:52:54.054 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T21:52:54.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 408021893137 -lt 408021893137 2026-02-22T21:52:54.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:52:54.253 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 10-214748364839 2026-02-22T21:52:54.253 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:52:54.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=10 2026-02-22T21:52:54.254 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 10-214748364839 2026-02-22T21:52:54.254 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:52:54.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=214748364839 2026-02-22T21:52:54.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.10 seq 214748364839' 2026-02-22T21:52:54.256 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.10 seq 214748364839 2026-02-22T21:52:54.257 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 10 2026-02-22T21:52:54.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 214748364840 -lt 214748364839 2026-02-22T21:52:54.452 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:52:54.453 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:52:54.453 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:52:54.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 28 == 0 2026-02-22T21:52:54.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:52:54.715 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:52:54.715 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:52:54.715 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:52:54.715 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:52:54.716 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:52:54.716 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:52:54.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=28 2026-02-22T21:52:54.916 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:52:54.916 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:52:54.916 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:52:55.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 28 = 28 2026-02-22T21:52:55.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:52:55.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:52:55.183 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2026-02-22T21:52:55.183 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:52:55.183 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:52:55.184 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2026-02-22T21:52:55.184 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:52:55.379 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2026-02-22T21:52:55.379 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:52:55.380 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:52:55.380 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:52:55.380 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:52:55.380 INFO:tasks.workunit.client.0.trial034.stderr:10' 2026-02-22T21:52:55.380 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2026-02-22T21:52:55.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2026-02-22T21:52:55.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2026-02-22T21:52:55.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:229: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2026-02-22T21:52:55.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-jerasure 2026-02-22T21:52:55.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T21:52:55.593 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T21:52:55.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2026-02-22T21:52:55.850 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile profile-jerasure does not exist 2026-02-22T21:52:55.860 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-22T21:52:55.860 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2026-02-22T21:52:55.860 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2026-02-22T21:52:55.860 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:157: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2026-02-22T21:52:55.860 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:158: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2026-02-22T21:52:55.860 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:160: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2026-02-22T21:52:56.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:165: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2026-02-22T21:52:56.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2026-02-22T21:52:56.525 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-lrc-a' already exists 2026-02-22T21:52:56.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:52:57.538 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:168: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2026-02-22T21:52:57.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-22T21:52:57.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2026-02-22T21:52:57.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-22T21:52:57.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:57.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-22T21:52:57.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:57.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-22T21:52:57.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:57.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-22T21:52:57.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:52:57.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-22T21:52:57.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-22T21:52:57.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:52:57.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:52:57.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-22T21:52:57.591 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2026-02-22T21:52:57.591 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2026-02-22T21:52:57.591 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:52:57.591 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2026-02-22T21:52:57.591 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:52:57.792 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2026-02-22T21:52:57.792 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:52:57.792 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T21:52:57.792 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 3 2026-02-22T21:52:57.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '9' '3') 2026-02-22T21:52:57.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-22T21:52:57.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2026-02-22T21:52:57.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2026-02-22T21:52:58.038 INFO:tasks.workunit.client.0.trial034.stderr:osd.3 is already out. 2026-02-22T21:52:58.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-22T21:53:03.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc-a SOMETHING 2026-02-22T21:53:03.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2026-02-22T21:53:03.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:53:03.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<3\>' 2026-02-22T21:53:03.050 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2026-02-22T21:53:03.050 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:53:03.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2026-02-22T21:53:03.245 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:53:03.245 INFO:tasks.workunit.client.0.trial034.stderr:7' 2026-02-22T21:53:03.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 7 2026-02-22T21:53:03.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:53:03.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:53:03.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 3 2026-02-22T21:53:03.509 INFO:tasks.workunit.client.0.trial034.stderr:osd.3 is already in. 2026-02-22T21:53:03.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-22T21:53:03.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:170: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2026-02-22T21:53:03.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc-a 2026-02-22T21:53:03.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2026-02-22T21:53:03.711 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-lrc-a' does not exist 2026-02-22T21:53:03.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2026-02-22T21:53:04.019 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile profile-lrc-a does not exist 2026-02-22T21:53:04.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-22T21:53:04.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2026-02-22T21:53:04.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2026-02-22T21:53:04.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:176: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2026-02-22T21:53:04.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:177: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2026-02-22T21:53:04.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2026-02-22T21:53:04.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:183: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2026-02-22T21:53:04.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2026-02-22T21:53:04.638 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-lrc' already exists 2026-02-22T21:53:04.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:186: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-22T21:53:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-22T21:53:05.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:53:05.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:53:05.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-22T21:53:05.694 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2026-02-22T21:53:05.694 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2026-02-22T21:53:05.694 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:53:05.695 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2026-02-22T21:53:05.695 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:53:05.890 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2026-02-22T21:53:05.890 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:53:05.890 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:53:05.891 INFO:tasks.workunit.client.0.trial034.stderr:10 2026-02-22T21:53:05.891 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:53:05.891 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:53:05.891 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:53:05.891 INFO:tasks.workunit.client.0.trial034.stderr:7' 2026-02-22T21:53:05.891 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 7 2026-02-22T21:53:05.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('8' '6' '0' '10' '5' '3' '9' '7') 2026-02-22T21:53:05.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-22T21:53:05.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2026-02-22T21:53:05.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2026-02-22T21:53:06.145 INFO:tasks.workunit.client.0.trial034.stderr:osd.7 is already out. 2026-02-22T21:53:06.156 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-22T21:53:11.158 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc SOMETHING 2026-02-22T21:53:11.158 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2026-02-22T21:53:11.158 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:53:11.159 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2026-02-22T21:53:11.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2026-02-22T21:53:11.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:53:11.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2026-02-22T21:53:11.359 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:53:11.359 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:53:11.359 INFO:tasks.workunit.client.0.trial034.stderr:10 2026-02-22T21:53:11.359 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:53:11.359 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:53:11.359 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:53:11.359 INFO:tasks.workunit.client.0.trial034.stderr:4' 2026-02-22T21:53:11.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 4 2026-02-22T21:53:11.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:53:11.378 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:53:11.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2026-02-22T21:53:11.628 INFO:tasks.workunit.client.0.trial034.stderr:osd.7 is already in. 2026-02-22T21:53:11.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-22T21:53:11.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:188: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2026-02-22T21:53:11.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc 2026-02-22T21:53:11.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2026-02-22T21:53:11.927 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-lrc' does not exist 2026-02-22T21:53:11.937 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2026-02-22T21:53:12.186 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile profile-lrc does not exist 2026-02-22T21:53:12.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2026-02-22T21:53:12.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2026-02-22T21:53:12.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:234: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2026-02-22T21:53:12.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:236: TEST_rados_put_get_shec: local poolname=pool-shec 2026-02-22T21:53:12.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:237: TEST_rados_put_get_shec: local profile=profile-shec 2026-02-22T21:53:12.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2026-02-22T21:53:12.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:243: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2026-02-22T21:53:12.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2026-02-22T21:53:12.817 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-shec' already exists 2026-02-22T21:53:12.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:53:13.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:246: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2026-02-22T21:53:13.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2026-02-22T21:53:13.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2026-02-22T21:53:13.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2026-02-22T21:53:13.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:53:13.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2026-02-22T21:53:13.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:53:13.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2026-02-22T21:53:13.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:53:13.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2026-02-22T21:53:13.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2026-02-22T21:53:13.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2026-02-22T21:53:13.834 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2026-02-22T21:53:13.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:53:13.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:53:13.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2026-02-22T21:53:13.882 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2026-02-22T21:53:13.882 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2026-02-22T21:53:13.882 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:53:13.883 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2026-02-22T21:53:13.883 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:53:14.083 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2026-02-22T21:53:14.084 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:53:14.084 INFO:tasks.workunit.client.0.trial034.stderr:1' 2026-02-22T21:53:14.084 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 1 2026-02-22T21:53:14.084 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('7' '9' '1') 2026-02-22T21:53:14.084 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2026-02-22T21:53:14.084 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2026-02-22T21:53:14.084 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2026-02-22T21:53:14.330 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 is already out. 2026-02-22T21:53:14.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2026-02-22T21:53:19.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-shec SOMETHING 2026-02-22T21:53:19.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2026-02-22T21:53:19.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2026-02-22T21:53:19.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<1\>' 2026-02-22T21:53:19.342 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2026-02-22T21:53:19.342 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:53:19.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2026-02-22T21:53:19.537 INFO:tasks.workunit.client.0.trial034.stderr:9 2026-02-22T21:53:19.537 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T21:53:19.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 6 2026-02-22T21:53:19.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2026-02-22T21:53:19.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2026-02-22T21:53:19.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 1 2026-02-22T21:53:19.807 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 is already in. 2026-02-22T21:53:19.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2026-02-22T21:53:19.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:248: TEST_rados_put_get_shec: delete_pool pool-shec 2026-02-22T21:53:19.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-shec 2026-02-22T21:53:19.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2026-02-22T21:53:20.110 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-shec' does not exist 2026-02-22T21:53:20.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2026-02-22T21:53:20.363 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile profile-shec does not exist 2026-02-22T21:53:20.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2026-02-22T21:53:20.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=ecpool 2026-02-22T21:53:20.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2026-02-22T21:53:20.622 INFO:tasks.workunit.client.0.trial034.stderr:pool 'ecpool' does not exist 2026-02-22T21:53:20.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2026-02-22T21:53:20.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2026-02-22T21:53:20.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:53:20.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2026-02-22T21:53:20.632 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:53:20.632 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:53:20.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:53:20.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:53:20.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:53:20.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:53:20.746 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:53:20.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:53:20.747 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:53:20.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:53:20.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:53:20.749 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:53:20.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:53:20.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:53:20.751 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:53:20.751 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:53:20.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:53:20.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:53:20.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2026-02-22T21:53:20.818 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:53:20.818 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:20.818 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:53:20.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.42545 2026-02-22T21:53:20.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:53:20.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:53:20.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2377: main: code=0 2026-02-22T21:53:20.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: main: teardown td/test-erasure-code 0 2026-02-22T21:53:20.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2026-02-22T21:53:20.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2026-02-22T21:53:20.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2026-02-22T21:53:20.821 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:53:20.821 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:53:20.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:53:20.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:53:20.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:53:20.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:53:20.823 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:53:20.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:53:20.824 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:53:20.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:53:20.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:53:20.825 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:53:20.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:53:20.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:53:20.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:53:20.827 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:53:20.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:53:20.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T21:53:20.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2026-02-22T21:53:20.830 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:53:20.830 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:20.830 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.42545 2026-02-22T21:53:20.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.42545 2026-02-22T21:53:20.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:53:20.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:53:20.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2382: main: return 0 2026-02-22T21:53:20.833 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T21:53:20.833 DEBUG:teuthology.orchestra.run.trial034:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T21:53:20.883 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2026-02-22T21:53:20.883 DEBUG:teuthology.orchestra.run.trial034:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f90f4eaebb28b619cf76e80cbbe997656da69579 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2026-02-22T21:53:20.936 INFO:tasks.workunit.client.0.trial034.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: export PYTHONWARNINGS=ignore 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: main: PYTHONWARNINGS=ignore 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: export CEPH_CONF=/dev/null 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: main: CEPH_CONF=/dev/null 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: main: unset CEPH_ARGS 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: main: local code 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: main: run td/test-erasure-eio 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2026-02-22T21:53:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2026-02-22T21:53:20.940 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2026-02-22T21:53:20.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none ' 2026-02-22T21:53:20.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2026-02-22T21:53:20.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2026-02-22T21:53:20.941 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2026-02-22T21:53:20.941 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T21:53:20.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2026-02-22T21:53:20.943 INFO:tasks.workunit.client.0.trial034.stderr:TEST_ec_object_attr_read_error 2026-02-22T21:53:20.943 INFO:tasks.workunit.client.0.trial034.stderr:TEST_ec_recovery_multiple_errors 2026-02-22T21:53:20.943 INFO:tasks.workunit.client.0.trial034.stderr:TEST_ec_recovery_multiple_objects 2026-02-22T21:53:20.943 INFO:tasks.workunit.client.0.trial034.stderr:TEST_ec_recovery_multiple_objects_eio 2026-02-22T21:53:20.943 INFO:tasks.workunit.client.0.trial034.stderr:TEST_ec_recovery_unfound 2026-02-22T21:53:20.943 INFO:tasks.workunit.client.0.trial034.stderr:TEST_ec_single_recovery_error 2026-02-22T21:53:20.943 INFO:tasks.workunit.client.0.trial034.stderr:TEST_rados_get_bad_size_shard_0 2026-02-22T21:53:20.943 INFO:tasks.workunit.client.0.trial034.stderr:TEST_rados_get_bad_size_shard_1 2026-02-22T21:53:20.943 INFO:tasks.workunit.client.0.trial034.stderr:TEST_rados_get_subread_eio_shard_0 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:TEST_rados_get_subread_eio_shard_1 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:TEST_rados_get_subread_missing 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:53:20.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:53:20.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:53:20.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:53:20.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:53:20.947 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:53:20.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:53:20.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:53:20.948 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:53:20.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:53:20.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:53:20.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:53:20.949 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:53:20.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:53:20.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:53:20.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T21:53:20.953 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:53:20.953 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:20.953 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:53:20.953 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T21:53:20.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:53:20.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:53:20.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T21:53:20.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:53:20.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:20.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:53:20.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T21:53:20.958 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:53:20.959 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2026-02-22T21:53:20.959 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2026-02-22T21:53:20.959 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:53:20.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T21:53:20.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T21:53:20.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T21:53:20.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:53:20.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:53:20.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:53:20.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T21:53:20.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T21:53:20.980 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:53:20.981 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:53:20.981 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:53:20.981 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:53:20.981 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:20.981 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:53:20.981 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:53:20.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T21:53:21.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:53:21.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:53:21.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:53:21.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:53:21.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:53:21.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:53:21.011 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:53:21.012 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:53:21.012 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:53:21.013 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:53:21.013 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:21.013 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:53:21.013 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:53:21.013 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:53:21.013 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T21:53:21.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:53:21.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:53:21.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:53:21.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:53:21.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:53:21.077 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:53:21.078 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:53:21.078 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:53:21.078 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:53:21.078 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:21.078 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:53:21.078 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:53:21.078 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:53:21.078 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T21:53:21.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T21:53:21.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T21:53:21.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:53:21.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:53:21.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:53:21.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T21:53:21.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:53:21.239 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:53:21.239 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:53:21.239 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:53:21.239 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:53:21.239 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:21.239 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:53:21.240 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:53:21.240 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:53:21.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:53:21.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T21:53:21.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T21:53:21.362 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T21:53:21.370 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:53:22.373 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T21:53:22.373 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:53:22.373 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:53:22.374 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:53:22.374 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:22.374 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:53:22.374 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:53:22.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T21:53:22.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T21:53:22.421 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:53:22.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T21:53:22.428 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:53:20.997+0000 7fbeaae1fec0 0 load: jerasure load: lrc 2026-02-22T21:53:22.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2026-02-22T21:53:22.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2026-02-22T21:53:22.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_backfill_unfound: local objname=myobject 2026-02-22T21:53:22.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_backfill_unfound: local lastobj=300 2026-02-22T21:53:22.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:515: TEST_ec_backfill_unfound: local testobj=obj250 2026-02-22T21:53:22.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:53:22.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:518: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-22T21:53:22.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:519: TEST_ec_backfill_unfound: setup_osds 5 2026-02-22T21:53:22.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2026-02-22T21:53:22.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T21:53:22.431 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2026-02-22T21:53:22.434 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2026-02-22T21:53:22.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:53:22.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T21:53:22.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:53:22.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:53:22.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:53:22.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:53:22.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T21:53:22.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-22T21:53:22.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:53:22.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:53:22.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:53:22.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T21:53:22.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T21:53:22.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:53:22.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:53:22.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:53:22.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:53:22.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:53:22.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:53:22.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:53:22.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:53:22.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:53:22.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:53:22.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T21:53:22.437 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:53:22.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=79023de3-07af-4222-a70c-07515ccf5657 2026-02-22T21:53:22.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 79023de3-07af-4222-a70c-07515ccf5657' 2026-02-22T21:53:22.438 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 79023de3-07af-4222-a70c-07515ccf5657 2026-02-22T21:53:22.439 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:53:22.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSeptpefu8GhAAZ1SqLMyzGr1NK4/QBHmPKQ== 2026-02-22T21:53:22.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSeptpefu8GhAAZ1SqLMyzGr1NK4/QBHmPKQ=="}' 2026-02-22T21:53:22.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 79023de3-07af-4222-a70c-07515ccf5657 -i td/test-erasure-eio/0/new.json 2026-02-22T21:53:22.557 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:53:22.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T21:53:22.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSeptpefu8GhAAZ1SqLMyzGr1NK4/QBHmPKQ== --osd-uuid 79023de3-07af-4222-a70c-07515ccf5657 2026-02-22T21:53:22.582 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:22.577+0000 7f647b7dc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:22.583 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:22.581+0000 7f647b7dc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:22.584 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:22.581+0000 7f647b7dc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:22.585 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:22.581+0000 7f647b7dc8c0 -1 bdev(0x55f7f0305c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:53:22.585 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:22.581+0000 7f647b7dc8c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T21:53:25.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T21:53:25.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:53:25.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:53:25.141 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:53:25.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:53:25.417 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:53:25.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:53:25.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:53:25.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:53:25.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:53:25.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:53:25.434 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:25.429+0000 7ff43eec98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:25.436 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:25.433+0000 7ff43eec98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:25.437 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:25.433+0000 7ff43eec98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:25.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:53:25.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:53:25.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:53:25.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:53:25.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:53:25.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:25.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:53:25.608 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:53:25.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:25.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:53:25.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:26.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:26.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:26.807 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:53:26.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:53:26.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:26.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:53:27.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:27.161 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:27.157+0000 7ff43eec98c0 -1 Falling back to public interface 2026-02-22T21:53:28.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:28.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:28.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:53:28.013 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:53:28.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:28.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:53:28.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:29.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:29.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:29.216 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:53:29.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:53:29.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:29.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:53:29.416 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:30.167 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:30.165+0000 7ff43eec98c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:53:30.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:30.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:30.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:53:30.418 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:53:30.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:30.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:53:30.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:31.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:31.620 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:53:31.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:31.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:53:31.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:31.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:53:31.821 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3513970906,v1:127.0.0.1:6803/3513970906] [v2:127.0.0.1:6804/3513970906,v1:127.0.0.1:6805/3513970906] exists,up 79023de3-07af-4222-a70c-07515ccf5657 2026-02-22T21:53:31.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:53:31.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:53:31.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:53:31.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:53:31.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T21:53:31.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:53:31.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:53:31.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T21:53:31.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:53:31.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T21:53:31.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-22T21:53:31.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:53:31.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:53:31.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:53:31.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T21:53:31.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T21:53:31.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:53:31.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:53:31.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:53:31.826 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:53:31.826 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:53:31.826 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:53:31.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:53:31.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:53:31.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:53:31.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:53:31.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T21:53:31.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:53:31.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f78f130-77e2-4967-8afd-24f6a963f33d 2026-02-22T21:53:31.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9f78f130-77e2-4967-8afd-24f6a963f33d' 2026-02-22T21:53:31.829 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 9f78f130-77e2-4967-8afd-24f6a963f33d 2026-02-22T21:53:31.829 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:53:31.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbeptpYh36MRAAsv/a7dkDDchgWlhQ0Di93g== 2026-02-22T21:53:31.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbeptpYh36MRAAsv/a7dkDDchgWlhQ0Di93g=="}' 2026-02-22T21:53:31.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f78f130-77e2-4967-8afd-24f6a963f33d -i td/test-erasure-eio/1/new.json 2026-02-22T21:53:32.044 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:53:32.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T21:53:32.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbeptpYh36MRAAsv/a7dkDDchgWlhQ0Di93g== --osd-uuid 9f78f130-77e2-4967-8afd-24f6a963f33d 2026-02-22T21:53:32.080 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:32.076+0000 7fdd763138c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:32.082 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:32.076+0000 7fdd763138c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:32.083 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:32.080+0000 7fdd763138c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:32.084 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:32.080+0000 7fdd763138c0 -1 bdev(0x56138d351c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:53:32.084 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:32.080+0000 7fdd763138c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T21:53:34.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T21:53:34.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:53:34.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:53:34.405 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T21:53:34.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:53:34.677 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T21:53:34.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T21:53:34.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:53:34.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:53:34.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:53:34.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:53:34.702 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:34.696+0000 7f2944d6b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:34.704 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:34.700+0000 7f2944d6b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:34.705 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:34.700+0000 7f2944d6b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:34.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T21:53:34.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:53:34.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T21:53:34.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:53:34.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:53:34.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:34.883 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:53:34.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:53:34.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:34.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:53:35.083 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:36.084 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:36.084 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:36.084 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:53:36.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:53:36.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:36.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:53:36.208 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:36.204+0000 7f2944d6b8c0 -1 Falling back to public interface 2026-02-22T21:53:36.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:37.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:37.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:37.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:53:37.286 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:53:37.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:37.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:53:37.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:38.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:38.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:38.487 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:53:38.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:53:38.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:38.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:53:38.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:39.450 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:39.444+0000 7f2944d6b8c0 -1 osd.1 0 log_to_monitors true 2026-02-22T21:53:39.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:39.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:39.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:53:39.687 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:53:39.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:39.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:53:39.886 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:40.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:40.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:40.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:53:40.888 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:53:40.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:40.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:53:41.092 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1464350240,v1:127.0.0.1:6811/1464350240] [v2:127.0.0.1:6812/1464350240,v1:127.0.0.1:6813/1464350240] exists,up 9f78f130-77e2-4967-8afd-24f6a963f33d 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:53:41.093 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:53:41.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:53:41.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:53:41.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:53:41.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:53:41.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T21:53:41.096 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:53:41.097 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=44dbedf4-066d-448d-bd05-d807a608b692 2026-02-22T21:53:41.097 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 44dbedf4-066d-448d-bd05-d807a608b692' 2026-02-22T21:53:41.097 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 44dbedf4-066d-448d-bd05-d807a608b692 2026-02-22T21:53:41.097 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:53:41.110 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDleptpaaegBhAAWW2sV5n6E9IyuMg1I4VK7g== 2026-02-22T21:53:41.110 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDleptpaaegBhAAWW2sV5n6E9IyuMg1I4VK7g=="}' 2026-02-22T21:53:41.110 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 44dbedf4-066d-448d-bd05-d807a608b692 -i td/test-erasure-eio/2/new.json 2026-02-22T21:53:41.308 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:53:41.317 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T21:53:41.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDleptpaaegBhAAWW2sV5n6E9IyuMg1I4VK7g== --osd-uuid 44dbedf4-066d-448d-bd05-d807a608b692 2026-02-22T21:53:41.334 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:41.328+0000 7fa78a5b38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:41.335 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:41.332+0000 7fa78a5b38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:41.337 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:41.332+0000 7fa78a5b38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:41.337 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:41.332+0000 7fa78a5b38c0 -1 bdev(0x55b8872e9c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:53:41.337 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:41.332+0000 7fa78a5b38c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T21:53:43.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T21:53:43.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:53:43.669 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T21:53:43.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:53:43.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:53:43.934 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T21:53:43.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T21:53:43.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:53:43.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:53:43.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:53:43.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:53:43.960 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:43.956+0000 7fa3ab5808c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:43.962 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:43.956+0000 7fa3ab5808c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:43.963 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:43.960+0000 7fa3ab5808c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:44.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T21:53:44.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:53:44.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T21:53:44.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:53:44.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:53:44.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:44.132 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:53:44.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:53:44.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:44.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:53:44.331 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:45.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:45.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:45.333 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:53:45.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:53:45.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:45.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:53:45.533 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:45.688 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:45.684+0000 7fa3ab5808c0 -1 Falling back to public interface 2026-02-22T21:53:46.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:46.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:46.535 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:53:46.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:53:46.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:46.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:53:46.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:47.735 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:47.735 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:47.735 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:53:47.735 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:53:47.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:47.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:53:47.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:48.708 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:48.704+0000 7fa3ab5808c0 -1 osd.2 0 log_to_monitors true 2026-02-22T21:53:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:53:48.933 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:53:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:53:49.130 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:50.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:50.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:50.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:53:50.133 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:53:50.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:50.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:53:50.331 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2797805370,v1:127.0.0.1:6819/2797805370] [v2:127.0.0.1:6820/2797805370,v1:127.0.0.1:6821/2797805370] exists,up 44dbedf4-066d-448d-bd05-d807a608b692 2026-02-22T21:53:50.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:53:50.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:53:50.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:53:50.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:53:50.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:53:50.333 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:50.334 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:53:50.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T21:53:50.335 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:53:50.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7edba348-aa5f-4044-8d41-76046a8c5f90 2026-02-22T21:53:50.336 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 7edba348-aa5f-4044-8d41-76046a8c5f90 2026-02-22T21:53:50.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 7edba348-aa5f-4044-8d41-76046a8c5f90' 2026-02-22T21:53:50.336 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:53:50.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDueptpaWjXFBAAD3mHYv3ZzYLRT+SA4D8SRw== 2026-02-22T21:53:50.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDueptpaWjXFBAAD3mHYv3ZzYLRT+SA4D8SRw=="}' 2026-02-22T21:53:50.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7edba348-aa5f-4044-8d41-76046a8c5f90 -i td/test-erasure-eio/3/new.json 2026-02-22T21:53:50.553 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:53:50.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T21:53:50.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDueptpaWjXFBAAD3mHYv3ZzYLRT+SA4D8SRw== --osd-uuid 7edba348-aa5f-4044-8d41-76046a8c5f90 2026-02-22T21:53:50.583 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:50.580+0000 7f14912228c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:50.584 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:50.580+0000 7f14912228c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:50.585 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:50.580+0000 7f14912228c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:50.585 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:50.580+0000 7f14912228c0 -1 bdev(0x55e984105c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T21:53:50.586 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:50.580+0000 7f14912228c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T21:53:52.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T21:53:52.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:53:52.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T21:53:52.865 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T21:53:52.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:53:53.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T21:53:53.134 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T21:53:53.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:53:53.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:53:53.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:53:53.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:53:53.159 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:53.156+0000 7f6922ed98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:53.161 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:53.156+0000 7f6922ed98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:53.162 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:53.156+0000 7f6922ed98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:53.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T21:53:53.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:53:53.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T21:53:53.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:53:53.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:53:53.334 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:53:53.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:53.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:53:53.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:53.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:53:53.529 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:54.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:54.531 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:54.531 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:53:54.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:53:54.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:54.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:53:54.668 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:54.664+0000 7f6922ed98c0 -1 Falling back to public interface 2026-02-22T21:53:54.724 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:55.726 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:55.726 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:55.727 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:53:55.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:53:55.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:55.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:53:55.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:56.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:56.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:56.925 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:53:56.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:53:56.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:56.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:53:57.124 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:57.905 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:57.900+0000 7f6922ed98c0 -1 osd.3 0 log_to_monitors true 2026-02-22T21:53:58.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:58.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:58.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:53:58.126 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:53:58.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:58.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:53:58.365 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:53:59.366 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:53:59.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:53:59.367 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:53:59.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:53:59.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:53:59.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:53:59.565 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 27 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/124789163,v1:127.0.0.1:6827/124789163] [v2:127.0.0.1:6828/124789163,v1:127.0.0.1:6829/124789163] exists,up 7edba348-aa5f-4044-8d41-76046a8c5f90 2026-02-22T21:53:59.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:53:59.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:53:59.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:53:59.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:53:59.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-22T21:53:59.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:53:59.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:53:59.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2026-02-22T21:53:59.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:53:59.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-22T21:53:59.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-22T21:53:59.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:53:59.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:53:59.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:53:59.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-22T21:53:59.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-22T21:53:59.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:53:59.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:53:59.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:53:59.571 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:53:59.571 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:53:59.571 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:53:59.572 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:53:59.572 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:53:59.572 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:53:59.572 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:53:59.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:53:59.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:53:59.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:53:59.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:53:59.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:53:59.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:53:59.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:53:59.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:53:59.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:53:59.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:53:59.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:53:59.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:53:59.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-22T21:53:59.574 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:53:59.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a4cac79d-a400-4f87-a254-1f6b683908e6 2026-02-22T21:53:59.575 INFO:tasks.workunit.client.0.trial034.stdout:add osd4 a4cac79d-a400-4f87-a254-1f6b683908e6 2026-02-22T21:53:59.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 a4cac79d-a400-4f87-a254-1f6b683908e6' 2026-02-22T21:53:59.575 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:53:59.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3eptpIg7VIhAAPPoqjkvlJzmQaLumip5Tyg== 2026-02-22T21:53:59.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3eptpIg7VIhAAPPoqjkvlJzmQaLumip5Tyg=="}' 2026-02-22T21:53:59.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a4cac79d-a400-4f87-a254-1f6b683908e6 -i td/test-erasure-eio/4/new.json 2026-02-22T21:53:59.787 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:53:59.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-22T21:53:59.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3eptpIg7VIhAAPPoqjkvlJzmQaLumip5Tyg== --osd-uuid a4cac79d-a400-4f87-a254-1f6b683908e6 2026-02-22T21:53:59.813 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:59.808+0000 7f80aa0138c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:59.814 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:59.812+0000 7f80aa0138c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:59.815 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:59.812+0000 7f80aa0138c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:53:59.816 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:59.812+0000 7f80aa0138c0 -1 bdev(0x561be0fc1c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-22T21:53:59.816 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:53:59.812+0000 7f80aa0138c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-22T21:54:02.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-22T21:54:02.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:54:02.100 INFO:tasks.workunit.client.0.trial034.stdout:adding osd4 key to auth repository 2026-02-22T21:54:02.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2026-02-22T21:54:02.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:54:02.367 INFO:tasks.workunit.client.0.trial034.stdout:start osd.4 2026-02-22T21:54:02.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2026-02-22T21:54:02.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:54:02.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:54:02.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:54:02.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:54:02.392 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:02.388+0000 7efe9a9518c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:54:02.393 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:02.388+0000 7efe9a9518c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:54:02.395 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:02.392+0000 7efe9a9518c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:54:02.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2026-02-22T21:54:02.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:54:02.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2026-02-22T21:54:02.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:54:02.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:54:02.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:02.568 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:54:02.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:54:02.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:02.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:54:02.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:54:03.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:54:03.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:03.768 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:54:03.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:54:03.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:03.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:54:03.868 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:03.864+0000 7efe9a9518c0 -1 Falling back to public interface 2026-02-22T21:54:03.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:54:04.966 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:54:04.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:04.967 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:54:04.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:54:04.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:04.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:54:05.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:54:06.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:54:06.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:06.166 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:54:06.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:54:06.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:06.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:54:06.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:54:06.871 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:06.864+0000 7efe9a9518c0 -1 osd.4 0 log_to_monitors true 2026-02-22T21:54:07.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:54:07.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:07.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:54:07.356 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:54:07.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:07.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:54:07.558 INFO:tasks.workunit.client.0.trial034.stdout:osd.4 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2129653826,v1:127.0.0.1:6835/2129653826] [v2:127.0.0.1:6836/2129653826,v1:127.0.0.1:6837/2129653826] exists,up a4cac79d-a400-4f87-a254-1f6b683908e6 2026-02-22T21:54:07.559 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:54:07.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:54:07.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:54:07.560 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T21:54:07.560 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:54:07.560 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:54:07.560 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:54:07.560 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:54:07.561 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:54:07.561 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T21:54:07.561 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T21:54:07.561 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T21:54:07.622 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:54:07.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T21:54:07.630 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:53:27.161+0000 7ff43eec98c0 0 load: jerasure load: lrc 2026-02-22T21:54:07.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:54:07.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2026-02-22T21:54:07.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2026-02-22T21:54:07.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T21:54:07.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T21:54:07.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-22T21:54:07.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T21:54:07.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-22T21:54:07.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T21:54:07.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-22T21:54:07.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T21:54:07.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T21:54:08.208 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T21:54:08.218 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:54:09.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T21:54:09.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:54:09.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:54:09.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:54:09.220 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:54:09.221 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:54:09.221 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:54:09.221 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:54:09.221 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:54:09.221 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:54:09.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:54:09.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:54:09.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:54:09.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:54:09.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:54:09.324 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:54:09.524 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:54:09.525 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:54:09.525 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:54:09.525 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:54:09.525 INFO:tasks.workunit.client.0.trial034.stderr:4' 2026-02-22T21:54:09.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:54:09.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:54:09.526 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:54:09.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803786 2026-02-22T21:54:09.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803786 2026-02-22T21:54:09.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803786' 2026-02-22T21:54:09.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:54:09.601 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:54:09.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607560 2026-02-22T21:54:09.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607560 2026-02-22T21:54:09.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803786 1-51539607560' 2026-02-22T21:54:09.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:54:09.677 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:54:09.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411334 2026-02-22T21:54:09.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411334 2026-02-22T21:54:09.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803786 1-51539607560 2-77309411334' 2026-02-22T21:54:09.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:54:09.753 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:54:09.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=115964116997 2026-02-22T21:54:09.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 115964116997 2026-02-22T21:54:09.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803786 1-51539607560 2-77309411334 3-115964116997' 2026-02-22T21:54:09.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:54:09.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:54:09.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=150323855363 2026-02-22T21:54:09.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 150323855363 2026-02-22T21:54:09.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803786 1-51539607560 2-77309411334 3-115964116997 4-150323855363' 2026-02-22T21:54:09.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:54:09.905 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803786 2026-02-22T21:54:09.905 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:54:09.906 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:54:09.907 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803786 2026-02-22T21:54:09.907 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:54:09.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803786 2026-02-22T21:54:09.908 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803786 2026-02-22T21:54:09.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803786' 2026-02-22T21:54:09.909 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:54:10.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803784 -lt 25769803786 2026-02-22T21:54:10.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:54:11.122 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:54:11.122 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:54:11.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803784 -lt 25769803786 2026-02-22T21:54:11.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:54:12.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T21:54:12.321 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:54:12.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803787 -lt 25769803786 2026-02-22T21:54:12.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:54:12.531 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607560 2026-02-22T21:54:12.531 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:54:12.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:54:12.533 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607560 2026-02-22T21:54:12.533 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:54:12.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607560 2026-02-22T21:54:12.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607560' 2026-02-22T21:54:12.535 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607560 2026-02-22T21:54:12.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:54:12.732 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607561 -lt 51539607560 2026-02-22T21:54:12.732 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:54:12.732 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411334 2026-02-22T21:54:12.732 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:54:12.734 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:54:12.734 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411334 2026-02-22T21:54:12.735 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:54:12.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411334 2026-02-22T21:54:12.736 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 77309411334 2026-02-22T21:54:12.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411334' 2026-02-22T21:54:12.737 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:54:12.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411335 -lt 77309411334 2026-02-22T21:54:12.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:54:12.947 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-115964116997 2026-02-22T21:54:12.947 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:54:12.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:54:12.949 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-115964116997 2026-02-22T21:54:12.949 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:54:12.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=115964116997 2026-02-22T21:54:12.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 115964116997' 2026-02-22T21:54:12.951 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 115964116997 2026-02-22T21:54:12.952 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:54:13.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 115964116997 -lt 115964116997 2026-02-22T21:54:13.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:54:13.147 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-150323855363 2026-02-22T21:54:13.147 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:54:13.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:54:13.149 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-150323855363 2026-02-22T21:54:13.149 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:54:13.151 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=150323855363 2026-02-22T21:54:13.152 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 150323855363 2026-02-22T21:54:13.152 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 150323855363' 2026-02-22T21:54:13.153 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:54:13.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 150323855363 -lt 150323855363 2026-02-22T21:54:13.349 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:54:13.350 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:54:13.350 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:54:13.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T21:54:13.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:54:13.612 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:54:13.612 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:54:13.612 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:54:13.612 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:54:13.612 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:54:13.612 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:54:13.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T21:54:13.814 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:54:13.814 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:54:13.814 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:54:14.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T21:54:14.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:54:14.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:54:14.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:525: TEST_ec_backfill_unfound: ceph pg dump pgs 2026-02-22T21:54:14.259 INFO:tasks.workunit.client.0.trial034.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T21:54:14.259 INFO:tasks.workunit.client.0.trial034.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:09.357370+0000 0'0 42:15 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-22T21:54:08.162270+0000 0'0 2026-02-22T21:54:08.162270+0000 0 0 periodic scrub scheduled @ 2026-02-24T04:24:58.900321+0000 0 0 2026-02-22T21:54:14.259 INFO:tasks.workunit.client.0.trial034.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:07.976096+0000 0'0 42:20 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-24T04:24:21.069412+0000 0 0 2026-02-22T21:54:14.260 INFO:tasks.workunit.client.0.trial034.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:07.976393+0000 0'0 42:99 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-24T03:45:30.196256+0000 0 0 2026-02-22T21:54:14.260 INFO:tasks.workunit.client.0.trial034.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:07.975874+0000 0'0 42:20 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-23T22:15:53.951714+0000 0 0 2026-02-22T21:54:14.260 INFO:tasks.workunit.client.0.trial034.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:53:49.839543+0000 0'0 41:64 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-24T05:45:36.906335+0000 0 0 2026-02-22T21:54:14.260 INFO:tasks.workunit.client.0.trial034.stdout: 2026-02-22T21:54:14.260 INFO:tasks.workunit.client.0.trial034.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T21:54:14.260 INFO:tasks.workunit.client.0.trial034.stderr:dumped pgs 2026-02-22T21:54:14.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:527: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-22T21:54:14.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T21:54:14.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T21:54:14.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-22T21:54:14.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:54:14.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T21:54:14.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:54:14.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T21:54:14.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:54:14.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T21:54:14.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:54:14.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T21:54:14.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:14.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2026-02-22T21:54:14.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2026-02-22T21:54:14.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2026-02-22T21:54:14.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2026-02-22T21:54:14.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2026-02-22T21:54:14.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: local primary=3 2026-02-22T21:54:14.490 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2026-02-22T21:54:14.490 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:54:14.490 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2026-02-22T21:54:14.490 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-22T21:54:14.491 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:54:14.691 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T21:54:14.691 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:54:14.691 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:54:14.691 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:54:14.691 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T21:54:14.691 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2026-02-22T21:54:14.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: initial_osds=('3' '1' '4' '0' '2') 2026-02-22T21:54:14.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: local -a initial_osds 2026-02-22T21:54:14.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local last_osd=2 2026-02-22T21:54:14.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-22T21:54:14.693 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:54:14.693 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:54:14.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:54:14.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:54:14.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:54:16.002 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:54:16.002 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: ceph osd down 2 2026-02-22T21:54:16.194 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 is already down. 2026-02-22T21:54:16.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:534: TEST_ec_backfill_unfound: ceph osd out 2 2026-02-22T21:54:16.453 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 is already out. 2026-02-22T21:54:16.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:536: TEST_ec_backfill_unfound: ceph pg dump pgs 2026-02-22T21:54:16.659 INFO:tasks.workunit.client.0.trial034.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T21:54:16.659 INFO:tasks.workunit.client.0.trial034.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:09.357370+0000 0'0 42:15 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-22T21:54:08.162270+0000 0'0 2026-02-22T21:54:08.162270+0000 0 0 periodic scrub scheduled @ 2026-02-24T04:24:58.900321+0000 0 0 2026-02-22T21:54:16.659 INFO:tasks.workunit.client.0.trial034.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:07.976096+0000 0'0 42:20 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-24T04:24:21.069412+0000 0 0 2026-02-22T21:54:16.659 INFO:tasks.workunit.client.0.trial034.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:07.976393+0000 0'0 44:103 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-24T03:45:30.196256+0000 0 0 2026-02-22T21:54:16.659 INFO:tasks.workunit.client.0.trial034.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:07.975874+0000 0'0 42:20 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-23T22:15:53.951714+0000 0 0 2026-02-22T21:54:16.659 INFO:tasks.workunit.client.0.trial034.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2026-02-22T21:54:14.891145+0000 0'0 44:73 [1,0] 1 [1,0] 1 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 no scrub is scheduled 0 0 2026-02-22T21:54:16.660 INFO:tasks.workunit.client.0.trial034.stdout: 2026-02-22T21:54:16.660 INFO:tasks.workunit.client.0.trial034.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T21:54:16.660 INFO:tasks.workunit.client.0.trial034.stderr:dumped pgs 2026-02-22T21:54:16.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2026-02-22T21:54:16.671 INFO:tasks.workunit.client.0.trial034.stderr:4+0 records in 2026-02-22T21:54:16.671 INFO:tasks.workunit.client.0.trial034.stderr:4+0 records out 2026-02-22T21:54:16.671 INFO:tasks.workunit.client.0.trial034.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000114994 s, 35.6 MB/s 2026-02-22T21:54:16.671 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: seq 1 300 2026-02-22T21:54:16.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.839 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.839 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.916 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:16.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:16.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.172 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.172 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.365 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.365 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.384 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.384 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.432 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.432 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.483 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.483 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.906 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.906 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:17.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:17.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.113 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.113 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.531 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.627 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.627 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:18.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:18.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.317 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.317 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:19.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:19.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.110 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.110 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.495 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.495 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.561 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.561 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.938 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:20.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:20.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.239 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.645 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.645 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:21.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:21.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.317 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.478 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.479 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.643 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.643 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.937 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.937 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:22.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:22.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.143 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.143 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.169 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:54:23.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2026-02-22T21:54:23.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:544: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2026-02-22T21:54:23.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T21:54:23.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T21:54:23.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T21:54:23.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T21:54:23.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T21:54:23.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T21:54:23.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=obj250 2026-02-22T21:54:23.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T21:54:23.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T21:54:23.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T21:54:23.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=0 2026-02-22T21:54:23.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T21:54:23.823 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure obj250 2026-02-22T21:54:23.823 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:54:23.823 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2026-02-22T21:54:23.823 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2026-02-22T21:54:23.823 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:54:24.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T21:54:24.016 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:54:24.016 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:54:24.016 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:54:24.016 INFO:tasks.workunit.client.0.trial034.stderr:2147483647' 2026-02-22T21:54:24.016 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2026-02-22T21:54:24.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2026-02-22T21:54:24.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T21:54:24.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=3 2026-02-22T21:54:24.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T21:54:24.016 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/3/type 2026-02-22T21:54:24.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T21:54:24.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-22T21:54:24.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T21:54:24.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2026-02-22T21:54:24.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T21:54:24.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T21:54:24.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T21:54:24.019 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2026-02-22T21:54:24.019 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T21:54:24.019 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T21:54:24.019 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:54:24.019 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:54:24.019 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:54:24.019 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T21:54:24.020 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-22T21:54:24.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T21:54:24.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T21:54:24.083 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T21:54:24.083 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.3 2026-02-22T21:54:24.083 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T21:54:24.083 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T21:54:24.083 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:54:24.084 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:54:24.084 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:54:24.084 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T21:54:24.084 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T21:54:24.084 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2026-02-22T21:54:24.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2026-02-22T21:54:24.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T21:54:24.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T21:54:24.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T21:54:24.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T21:54:24.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T21:54:24.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T21:54:24.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=obj250 2026-02-22T21:54:24.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T21:54:24.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T21:54:24.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T21:54:24.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=1 2026-02-22T21:54:24.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T21:54:24.143 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure obj250 2026-02-22T21:54:24.143 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:54:24.143 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2026-02-22T21:54:24.143 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2026-02-22T21:54:24.143 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:54:24.343 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T21:54:24.343 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:54:24.343 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:54:24.344 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:54:24.344 INFO:tasks.workunit.client.0.trial034.stderr:2147483647' 2026-02-22T21:54:24.344 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2026-02-22T21:54:24.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2026-02-22T21:54:24.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T21:54:24.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=1 2026-02-22T21:54:24.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T21:54:24.344 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/1/type 2026-02-22T21:54:24.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T21:54:24.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-22T21:54:24.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T21:54:24.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2026-02-22T21:54:24.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T21:54:24.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T21:54:24.347 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T21:54:24.347 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2026-02-22T21:54:24.347 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2026-02-22T21:54:24.347 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2026-02-22T21:54:24.347 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:54:24.347 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:54:24.347 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:54:24.347 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.1.asok 2026-02-22T21:54:24.347 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-22T21:54:24.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T21:54:24.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T21:54:24.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T21:54:24.408 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.1 2026-02-22T21:54:24.408 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2026-02-22T21:54:24.408 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2026-02-22T21:54:24.408 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:54:24.408 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:54:24.408 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:54:24.409 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.1.asok 2026-02-22T21:54:24.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T21:54:24.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2026-02-22T21:54:24.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2026-02-22T21:54:24.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T21:54:24.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T21:54:24.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T21:54:24.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T21:54:24.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T21:54:24.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:54:24.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:54:24.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:54:24.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:54:24.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T21:54:24.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T21:54:24.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:54:24.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T21:54:24.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:54:24.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T21:54:24.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:54:24.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:54:24.470 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:54:24.470 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T21:54:24.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T21:54:24.472 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T21:54:24.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T21:54:24.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:54:24.473 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-22T21:54:24.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T21:54:24.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T21:54:24.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T21:54:24.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T21:54:24.487 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:24.488+0000 7f28823168c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:54:24.488 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:24.488+0000 7f28823168c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:54:24.489 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:24.488+0000 7f28823168c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:54:24.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T21:54:24.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:54:24.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T21:54:24.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:54:24.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:54:24.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:24.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:54:24.667 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:54:24.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:24.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:54:24.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:54:25.764 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:25.764+0000 7f28823168c0 -1 Falling back to public interface 2026-02-22T21:54:25.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:54:25.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:25.862 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:54:25.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:54:25.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:25.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:54:26.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:54:27.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:54:27.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:27.067 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:54:27.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:54:27.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:27.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:54:27.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:54:28.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:54:28.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:28.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:54:28.265 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:54:28.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:28.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:54:28.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:54:28.865 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:28.864+0000 7f28823168c0 -1 osd.2 42 log_to_monitors true 2026-02-22T21:54:29.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:54:29.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:29.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:54:29.462 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:54:29.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:29.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:54:29.656 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up out weight 0 up_from 48 up_thru 20 down_at 43 last_clean_interval [18,42) [v2:127.0.0.1:6818/422608047,v1:127.0.0.1:6819/422608047] [v2:127.0.0.1:6820/422608047,v1:127.0.0.1:6821/422608047] exists,up 44dbedf4-066d-448d-bd05-d807a608b692 2026-02-22T21:54:29.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:54:29.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:54:29.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:54:29.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: ceph osd in 2 2026-02-22T21:54:29.921 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 is already in. 2026-02-22T21:54:29.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: sleep 15 2026-02-22T21:54:44.932 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: seq 1 240 2026-02-22T21:54:44.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2026-02-22T21:54:44.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2026-02-22T21:54:44.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2026-02-22T21:54:44.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2026-02-22T21:54:44.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2026-02-22T21:54:44.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2026-02-22T21:54:45.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2026-02-22T21:54:45.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2026-02-22T21:54:45.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2026-02-22T21:54:45.133 INFO:tasks.workunit.client.0.trial034.stdout:active+backfill_unfound+undersized+degraded+remapped 2026-02-22T21:54:45.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2026-02-22T21:54:45.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: break 2026-02-22T21:54:45.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:562: TEST_ec_backfill_unfound: ceph pg dump pgs 2026-02-22T21:54:45.329 INFO:tasks.workunit.client.0.trial034.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T21:54:45.329 INFO:tasks.workunit.client.0.trial034.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2026-02-22T21:54:31.228527+0000 46'301 54:905 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2026-02-22T21:54:08.162270+0000 0'0 2026-02-22T21:54:08.162270+0000 0 0 no scrub is scheduled 0 0 2026-02-22T21:54:45.330 INFO:tasks.workunit.client.0.trial034.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:29.973790+0000 0'0 54:58 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-24T01:44:43.378911+0000 0 0 2026-02-22T21:54:45.330 INFO:tasks.workunit.client.0.trial034.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:07.976393+0000 0'0 54:122 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-24T03:45:30.196256+0000 0 0 2026-02-22T21:54:45.330 INFO:tasks.workunit.client.0.trial034.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:07.975874+0000 0'0 54:43 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-23T22:15:53.951714+0000 0 0 2026-02-22T21:54:45.330 INFO:tasks.workunit.client.0.trial034.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:29.974076+0000 0'0 54:104 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-24T07:46:07.181062+0000 0 0 2026-02-22T21:54:45.330 INFO:tasks.workunit.client.0.trial034.stdout: 2026-02-22T21:54:45.330 INFO:tasks.workunit.client.0.trial034.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T21:54:45.330 INFO:tasks.workunit.client.0.trial034.stderr:dumped pgs 2026-02-22T21:54:45.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:563: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-22T21:54:45.338 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:54:45.338 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:54:45.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:54:45.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:54:45.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:54:46.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:54:46.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: sleep 5 2026-02-22T21:54:51.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:566: TEST_ec_backfill_unfound: ceph pg dump pgs 2026-02-22T21:54:51.831 INFO:tasks.workunit.client.0.trial034.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T21:54:51.832 INFO:tasks.workunit.client.0.trial034.stdout:2.0 301 1 303 0 1 1232896 0 0 101 200 101 active+recovery_unfound+undersized+degraded 2026-02-22T21:54:45.529176+0000 46'301 56:917 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2026-02-22T21:54:08.162270+0000 0'0 2026-02-22T21:54:08.162270+0000 0 0 no scrub is scheduled 0 0 2026-02-22T21:54:51.832 INFO:tasks.workunit.client.0.trial034.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2026-02-22T21:54:45.526837+0000 0'0 56:65 [4,1] 4 [4,1] 4 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 no scrub is scheduled 0 0 2026-02-22T21:54:51.832 INFO:tasks.workunit.client.0.trial034.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:07.976393+0000 0'0 56:126 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-24T03:45:30.196256+0000 0 0 2026-02-22T21:54:51.832 INFO:tasks.workunit.client.0.trial034.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T21:54:07.975874+0000 0'0 56:47 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 periodic scrub scheduled @ 2026-02-23T22:15:53.951714+0000 0 0 2026-02-22T21:54:51.832 INFO:tasks.workunit.client.0.trial034.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2026-02-22T21:54:45.526947+0000 0'0 56:111 [1,0] 1 [1,0] 1 0'0 2026-02-22T21:53:21.347161+0000 0'0 2026-02-22T21:53:21.347161+0000 0 0 no scrub is scheduled 0 0 2026-02-22T21:54:51.832 INFO:tasks.workunit.client.0.trial034.stdout: 2026-02-22T21:54:51.832 INFO:tasks.workunit.client.0.trial034.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T21:54:51.833 INFO:tasks.workunit.client.0.trial034.stderr:dumped pgs 2026-02-22T21:54:51.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-22T21:54:51.911 INFO:tasks.workunit.client.0.trial034.stdout:{ 2026-02-22T21:54:51.911 INFO:tasks.workunit.client.0.trial034.stdout: "num_missing": 1, 2026-02-22T21:54:51.911 INFO:tasks.workunit.client.0.trial034.stdout: "num_unfound": 1, 2026-02-22T21:54:51.911 INFO:tasks.workunit.client.0.trial034.stdout: "objects": [ 2026-02-22T21:54:51.911 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:51.912 INFO:tasks.workunit.client.0.trial034.stdout: "oid": { 2026-02-22T21:54:51.912 INFO:tasks.workunit.client.0.trial034.stdout: "oid": "obj250", 2026-02-22T21:54:51.912 INFO:tasks.workunit.client.0.trial034.stdout: "key": "", 2026-02-22T21:54:51.912 INFO:tasks.workunit.client.0.trial034.stdout: "snapid": -2, 2026-02-22T21:54:51.912 INFO:tasks.workunit.client.0.trial034.stdout: "hash": 2249616407, 2026-02-22T21:54:51.912 INFO:tasks.workunit.client.0.trial034.stdout: "max": 0, 2026-02-22T21:54:51.912 INFO:tasks.workunit.client.0.trial034.stdout: "pool": 2, 2026-02-22T21:54:51.912 INFO:tasks.workunit.client.0.trial034.stdout: "namespace": "" 2026-02-22T21:54:51.912 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:51.912 INFO:tasks.workunit.client.0.trial034.stdout: "need": "46'251", 2026-02-22T21:54:51.912 INFO:tasks.workunit.client.0.trial034.stdout: "have": "0'0", 2026-02-22T21:54:51.913 INFO:tasks.workunit.client.0.trial034.stdout: "flags": "none", 2026-02-22T21:54:51.913 INFO:tasks.workunit.client.0.trial034.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2026-02-22T21:54:51.913 INFO:tasks.workunit.client.0.trial034.stdout: "locations": [ 2026-02-22T21:54:51.913 INFO:tasks.workunit.client.0.trial034.stdout: "0(3)", 2026-02-22T21:54:51.913 INFO:tasks.workunit.client.0.trial034.stdout: "4(2)" 2026-02-22T21:54:51.913 INFO:tasks.workunit.client.0.trial034.stdout: ] 2026-02-22T21:54:51.913 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:51.913 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:51.913 INFO:tasks.workunit.client.0.trial034.stdout: "state": "NotRecovering", 2026-02-22T21:54:51.913 INFO:tasks.workunit.client.0.trial034.stdout: "available_might_have_unfound": true, 2026-02-22T21:54:51.913 INFO:tasks.workunit.client.0.trial034.stdout: "might_have_unfound": [ 2026-02-22T21:54:51.913 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:51.914 INFO:tasks.workunit.client.0.trial034.stdout: "osd": "2(4)", 2026-02-22T21:54:51.914 INFO:tasks.workunit.client.0.trial034.stdout: "status": "osd is down" 2026-02-22T21:54:51.914 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:51.914 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:51.914 INFO:tasks.workunit.client.0.trial034.stdout: "more": false 2026-02-22T21:54:51.914 INFO:tasks.workunit.client.0.trial034.stdout:} 2026-02-22T21:54:51.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph pg 2.0 query 2026-02-22T21:54:52.000 INFO:tasks.workunit.client.0.trial034.stdout:{ 2026-02-22T21:54:52.000 INFO:tasks.workunit.client.0.trial034.stdout: "snap_trimq": "[]", 2026-02-22T21:54:52.000 INFO:tasks.workunit.client.0.trial034.stdout: "snap_trimq_len": 0, 2026-02-22T21:54:52.001 INFO:tasks.workunit.client.0.trial034.stdout: "state": "active+recovery_unfound+undersized+degraded", 2026-02-22T21:54:52.001 INFO:tasks.workunit.client.0.trial034.stdout: "epoch": 56, 2026-02-22T21:54:52.001 INFO:tasks.workunit.client.0.trial034.stdout: "up": [ 2026-02-22T21:54:52.001 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T21:54:52.001 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T21:54:52.001 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T21:54:52.001 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T21:54:52.001 INFO:tasks.workunit.client.0.trial034.stdout: 2147483647 2026-02-22T21:54:52.001 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.001 INFO:tasks.workunit.client.0.trial034.stdout: "acting": [ 2026-02-22T21:54:52.001 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T21:54:52.001 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T21:54:52.002 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T21:54:52.002 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T21:54:52.002 INFO:tasks.workunit.client.0.trial034.stdout: 2147483647 2026-02-22T21:54:52.002 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.002 INFO:tasks.workunit.client.0.trial034.stdout: "acting_recovery_backfill": [ 2026-02-22T21:54:52.002 INFO:tasks.workunit.client.0.trial034.stdout: "0(3)", 2026-02-22T21:54:52.002 INFO:tasks.workunit.client.0.trial034.stdout: "1(1)", 2026-02-22T21:54:52.002 INFO:tasks.workunit.client.0.trial034.stdout: "3(0)", 2026-02-22T21:54:52.002 INFO:tasks.workunit.client.0.trial034.stdout: "4(2)" 2026-02-22T21:54:52.002 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.002 INFO:tasks.workunit.client.0.trial034.stdout: "info": { 2026-02-22T21:54:52.002 INFO:tasks.workunit.client.0.trial034.stdout: "pgid": "2.0s0", 2026-02-22T21:54:52.003 INFO:tasks.workunit.client.0.trial034.stdout: "shared": "0", 2026-02-22T21:54:52.003 INFO:tasks.workunit.client.0.trial034.stdout: "last_update": "46'301", 2026-02-22T21:54:52.003 INFO:tasks.workunit.client.0.trial034.stdout: "last_complete": "46'250", 2026-02-22T21:54:52.003 INFO:tasks.workunit.client.0.trial034.stdout: "log_tail": "46'200", 2026-02-22T21:54:52.003 INFO:tasks.workunit.client.0.trial034.stdout: "last_user_version": 301, 2026-02-22T21:54:52.003 INFO:tasks.workunit.client.0.trial034.stdout: "last_backfill": "MAX", 2026-02-22T21:54:52.003 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [], 2026-02-22T21:54:52.003 INFO:tasks.workunit.client.0.trial034.stdout: "history": { 2026-02-22T21:54:52.003 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_created": 40, 2026-02-22T21:54:52.003 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_pool_created": 40, 2026-02-22T21:54:52.003 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 56, 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_started": 55, 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 41, 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_clean": 40, 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_split": 0, 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_marked_full": 0, 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "same_up_since": 55, 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "same_interval_since": 55, 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "same_primary_since": 40, 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.004 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: "prior_readable_until_ub": 0 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: "stats": { 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: "version": "46'301", 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: "reported_seq": 918, 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: "reported_epoch": 56, 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: "state": "active+recovery_unfound+undersized+degraded", 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: "last_fresh": "2026-02-22T21:54:45.529176+0000", 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: "last_change": "2026-02-22T21:54:45.529176+0000", 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: "last_active": "2026-02-22T21:54:45.529176+0000", 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: "last_peered": "2026-02-22T21:54:45.529176+0000", 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean": "2026-02-22T21:54:14.292907+0000", 2026-02-22T21:54:52.005 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_active": "2026-02-22T21:54:45.527013+0000", 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_peered": "2026-02-22T21:54:45.527013+0000", 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "last_unstale": "2026-02-22T21:54:45.529176+0000", 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "last_undegraded": "2026-02-22T21:54:45.525354+0000", 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "last_fullsized": "2026-02-22T21:54:45.525202+0000", 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "mapping_epoch": 55, 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "log_start": "46'200", 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_start": "46'200", 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "created": 40, 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 41, 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "parent": "0.0", 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "parent_split_bits": 0, 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T21:54:52.006 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.007 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T21:54:52.007 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.007 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.007 INFO:tasks.workunit.client.0.trial034.stdout: "objects_scrubbed": 0, 2026-02-22T21:54:52.007 INFO:tasks.workunit.client.0.trial034.stdout: "log_size": 101, 2026-02-22T21:54:52.007 INFO:tasks.workunit.client.0.trial034.stdout: "log_dups_size": 200, 2026-02-22T21:54:52.007 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_size": 101, 2026-02-22T21:54:52.007 INFO:tasks.workunit.client.0.trial034.stdout: "stats_invalid": false, 2026-02-22T21:54:52.007 INFO:tasks.workunit.client.0.trial034.stdout: "dirty_stats_invalid": false, 2026-02-22T21:54:52.007 INFO:tasks.workunit.client.0.trial034.stdout: "omap_stats_invalid": false, 2026-02-22T21:54:52.007 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_stats_invalid": false, 2026-02-22T21:54:52.008 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_bytes_stats_invalid": false, 2026-02-22T21:54:52.008 INFO:tasks.workunit.client.0.trial034.stdout: "pin_stats_invalid": false, 2026-02-22T21:54:52.008 INFO:tasks.workunit.client.0.trial034.stdout: "manifest_stats_invalid": false, 2026-02-22T21:54:52.008 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrimq_len": 0, 2026-02-22T21:54:52.008 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_duration": 0, 2026-02-22T21:54:52.008 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_schedule": "no scrub is scheduled", 2026-02-22T21:54:52.008 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_duration": 0, 2026-02-22T21:54:52.008 INFO:tasks.workunit.client.0.trial034.stdout: "objects_trimmed": 0, 2026-02-22T21:54:52.008 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrim_duration": 0, 2026-02-22T21:54:52.008 INFO:tasks.workunit.client.0.trial034.stdout: "stat_sum": { 2026-02-22T21:54:52.009 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes": 1232896, 2026-02-22T21:54:52.009 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects": 301, 2026-02-22T21:54:52.009 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_clones": 0, 2026-02-22T21:54:52.009 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_copies": 1505, 2026-02-22T21:54:52.009 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing_on_primary": 1, 2026-02-22T21:54:52.009 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing": 1, 2026-02-22T21:54:52.009 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_degraded": 303, 2026-02-22T21:54:52.009 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_misplaced": 0, 2026-02-22T21:54:52.009 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_unfound": 1, 2026-02-22T21:54:52.009 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_dirty": 301, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_whiteouts": 0, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_read": 0, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_read_kb": 0, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_write": 301, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_write_kb": 1204, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_scrub_errors": 0, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_deep_scrub_errors": 0, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_recovered": 0, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_recovered": 0, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_keys_recovered": 0, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_omap": 0, 2026-02-22T21:54:52.010 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_kb": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_kb": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_promote": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_high": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_low": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_some": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_full": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_pinned": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_legacy_snapsets": 0, 2026-02-22T21:54:52.011 INFO:tasks.workunit.client.0.trial034.stdout: "num_large_omap_objects": 0, 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_manifest": 0, 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_bytes": 0, 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_keys": 0, 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_repaired": 0 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: "up": [ 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: 2147483647 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.012 INFO:tasks.workunit.client.0.trial034.stdout: "acting": [ 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: 2147483647 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: "avail_no_missing": [ 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: "0(3)", 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: "4(2)" 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: "object_location_counts": [ 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.013 INFO:tasks.workunit.client.0.trial034.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-22T21:54:52.014 INFO:tasks.workunit.client.0.trial034.stdout: "objects": 300 2026-02-22T21:54:52.014 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.014 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.014 INFO:tasks.workunit.client.0.trial034.stdout: "shards": "0(3),4(2)", 2026-02-22T21:54:52.014 INFO:tasks.workunit.client.0.trial034.stdout: "objects": 1 2026-02-22T21:54:52.014 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.014 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.014 INFO:tasks.workunit.client.0.trial034.stdout: "blocked_by": [], 2026-02-22T21:54:52.014 INFO:tasks.workunit.client.0.trial034.stdout: "up_primary": 3, 2026-02-22T21:54:52.014 INFO:tasks.workunit.client.0.trial034.stdout: "acting_primary": 3, 2026-02-22T21:54:52.014 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [] 2026-02-22T21:54:52.014 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.015 INFO:tasks.workunit.client.0.trial034.stdout: "empty": 0, 2026-02-22T21:54:52.015 INFO:tasks.workunit.client.0.trial034.stdout: "dne": 0, 2026-02-22T21:54:52.015 INFO:tasks.workunit.client.0.trial034.stdout: "incomplete": 0, 2026-02-22T21:54:52.015 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 56, 2026-02-22T21:54:52.015 INFO:tasks.workunit.client.0.trial034.stdout: "hit_set_history": { 2026-02-22T21:54:52.015 INFO:tasks.workunit.client.0.trial034.stdout: "current_last_update": "0'0", 2026-02-22T21:54:52.015 INFO:tasks.workunit.client.0.trial034.stdout: "history": [] 2026-02-22T21:54:52.015 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.015 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.015 INFO:tasks.workunit.client.0.trial034.stdout: "peer_info": [ 2026-02-22T21:54:52.015 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.016 INFO:tasks.workunit.client.0.trial034.stdout: "peer": "0(3)", 2026-02-22T21:54:52.016 INFO:tasks.workunit.client.0.trial034.stdout: "pgid": "2.0s3", 2026-02-22T21:54:52.016 INFO:tasks.workunit.client.0.trial034.stdout: "shared": "3", 2026-02-22T21:54:52.016 INFO:tasks.workunit.client.0.trial034.stdout: "last_update": "46'301", 2026-02-22T21:54:52.016 INFO:tasks.workunit.client.0.trial034.stdout: "last_complete": "46'301", 2026-02-22T21:54:52.016 INFO:tasks.workunit.client.0.trial034.stdout: "log_tail": "46'200", 2026-02-22T21:54:52.016 INFO:tasks.workunit.client.0.trial034.stdout: "last_user_version": 301, 2026-02-22T21:54:52.016 INFO:tasks.workunit.client.0.trial034.stdout: "last_backfill": "MAX", 2026-02-22T21:54:52.016 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [], 2026-02-22T21:54:52.016 INFO:tasks.workunit.client.0.trial034.stdout: "history": { 2026-02-22T21:54:52.017 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_created": 40, 2026-02-22T21:54:52.017 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_pool_created": 40, 2026-02-22T21:54:52.017 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 56, 2026-02-22T21:54:52.017 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_started": 55, 2026-02-22T21:54:52.017 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 41, 2026-02-22T21:54:52.017 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_clean": 40, 2026-02-22T21:54:52.017 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_split": 0, 2026-02-22T21:54:52.017 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_marked_full": 0, 2026-02-22T21:54:52.017 INFO:tasks.workunit.client.0.trial034.stdout: "same_up_since": 55, 2026-02-22T21:54:52.017 INFO:tasks.workunit.client.0.trial034.stdout: "same_interval_since": 55, 2026-02-22T21:54:52.017 INFO:tasks.workunit.client.0.trial034.stdout: "same_primary_since": 40, 2026-02-22T21:54:52.017 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T21:54:52.018 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.018 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T21:54:52.018 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.018 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.018 INFO:tasks.workunit.client.0.trial034.stdout: "prior_readable_until_ub": 0 2026-02-22T21:54:52.018 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.018 INFO:tasks.workunit.client.0.trial034.stdout: "stats": { 2026-02-22T21:54:52.018 INFO:tasks.workunit.client.0.trial034.stdout: "version": "46'301", 2026-02-22T21:54:52.018 INFO:tasks.workunit.client.0.trial034.stdout: "reported_seq": 908, 2026-02-22T21:54:52.018 INFO:tasks.workunit.client.0.trial034.stdout: "reported_epoch": 55, 2026-02-22T21:54:52.018 INFO:tasks.workunit.client.0.trial034.stdout: "state": "active+undersized+degraded", 2026-02-22T21:54:52.018 INFO:tasks.workunit.client.0.trial034.stdout: "last_fresh": "2026-02-22T21:54:23.820398+0000", 2026-02-22T21:54:52.019 INFO:tasks.workunit.client.0.trial034.stdout: "last_change": "2026-02-22T21:54:14.890666+0000", 2026-02-22T21:54:52.019 INFO:tasks.workunit.client.0.trial034.stdout: "last_active": "2026-02-22T21:54:23.820398+0000", 2026-02-22T21:54:52.019 INFO:tasks.workunit.client.0.trial034.stdout: "last_peered": "2026-02-22T21:54:23.820398+0000", 2026-02-22T21:54:52.019 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean": "2026-02-22T21:54:14.292907+0000", 2026-02-22T21:54:52.019 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_active": "2026-02-22T21:54:14.890666+0000", 2026-02-22T21:54:52.019 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_peered": "2026-02-22T21:54:14.890666+0000", 2026-02-22T21:54:52.019 INFO:tasks.workunit.client.0.trial034.stdout: "last_unstale": "2026-02-22T21:54:23.820398+0000", 2026-02-22T21:54:52.019 INFO:tasks.workunit.client.0.trial034.stdout: "last_undegraded": "2026-02-22T21:54:14.888829+0000", 2026-02-22T21:54:52.019 INFO:tasks.workunit.client.0.trial034.stdout: "last_fullsized": "2026-02-22T21:54:14.888736+0000", 2026-02-22T21:54:52.019 INFO:tasks.workunit.client.0.trial034.stdout: "mapping_epoch": 55, 2026-02-22T21:54:52.019 INFO:tasks.workunit.client.0.trial034.stdout: "log_start": "46'200", 2026-02-22T21:54:52.019 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_start": "46'200", 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "created": 40, 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 41, 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "parent": "0.0", 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "parent_split_bits": 0, 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "objects_scrubbed": 0, 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "log_size": 101, 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "log_dups_size": 200, 2026-02-22T21:54:52.020 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_size": 101, 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "stats_invalid": false, 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "dirty_stats_invalid": false, 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "omap_stats_invalid": false, 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_stats_invalid": false, 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_bytes_stats_invalid": false, 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "pin_stats_invalid": false, 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "manifest_stats_invalid": false, 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrimq_len": 0, 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_duration": 0, 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-24T00:19:14.591533+0000", 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_duration": 0, 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "objects_trimmed": 0, 2026-02-22T21:54:52.021 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrim_duration": 0, 2026-02-22T21:54:52.022 INFO:tasks.workunit.client.0.trial034.stdout: "stat_sum": { 2026-02-22T21:54:52.022 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes": 1232896, 2026-02-22T21:54:52.022 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects": 301, 2026-02-22T21:54:52.022 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_clones": 0, 2026-02-22T21:54:52.022 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_copies": 1505, 2026-02-22T21:54:52.022 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T21:54:52.022 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing": 0, 2026-02-22T21:54:52.022 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_degraded": 301, 2026-02-22T21:54:52.022 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_misplaced": 0, 2026-02-22T21:54:52.022 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_unfound": 0, 2026-02-22T21:54:52.022 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_dirty": 301, 2026-02-22T21:54:52.022 INFO:tasks.workunit.client.0.trial034.stdout: "num_whiteouts": 0, 2026-02-22T21:54:52.023 INFO:tasks.workunit.client.0.trial034.stdout: "num_read": 0, 2026-02-22T21:54:52.023 INFO:tasks.workunit.client.0.trial034.stdout: "num_read_kb": 0, 2026-02-22T21:54:52.023 INFO:tasks.workunit.client.0.trial034.stdout: "num_write": 301, 2026-02-22T21:54:52.023 INFO:tasks.workunit.client.0.trial034.stdout: "num_write_kb": 1204, 2026-02-22T21:54:52.023 INFO:tasks.workunit.client.0.trial034.stdout: "num_scrub_errors": 0, 2026-02-22T21:54:52.023 INFO:tasks.workunit.client.0.trial034.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T21:54:52.023 INFO:tasks.workunit.client.0.trial034.stdout: "num_deep_scrub_errors": 0, 2026-02-22T21:54:52.023 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_recovered": 0, 2026-02-22T21:54:52.023 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_recovered": 0, 2026-02-22T21:54:52.023 INFO:tasks.workunit.client.0.trial034.stdout: "num_keys_recovered": 0, 2026-02-22T21:54:52.024 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_omap": 0, 2026-02-22T21:54:52.024 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T21:54:52.024 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T21:54:52.024 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush": 0, 2026-02-22T21:54:52.024 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_kb": 0, 2026-02-22T21:54:52.024 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict": 0, 2026-02-22T21:54:52.024 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_kb": 0, 2026-02-22T21:54:52.024 INFO:tasks.workunit.client.0.trial034.stdout: "num_promote": 0, 2026-02-22T21:54:52.024 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_high": 0, 2026-02-22T21:54:52.024 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_low": 0, 2026-02-22T21:54:52.024 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_some": 0, 2026-02-22T21:54:52.024 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_full": 0, 2026-02-22T21:54:52.025 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_pinned": 0, 2026-02-22T21:54:52.025 INFO:tasks.workunit.client.0.trial034.stdout: "num_legacy_snapsets": 0, 2026-02-22T21:54:52.025 INFO:tasks.workunit.client.0.trial034.stdout: "num_large_omap_objects": 0, 2026-02-22T21:54:52.025 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_manifest": 0, 2026-02-22T21:54:52.025 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_bytes": 0, 2026-02-22T21:54:52.025 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_keys": 0, 2026-02-22T21:54:52.025 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_repaired": 0 2026-02-22T21:54:52.025 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.025 INFO:tasks.workunit.client.0.trial034.stdout: "up": [ 2026-02-22T21:54:52.025 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T21:54:52.026 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T21:54:52.026 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T21:54:52.026 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T21:54:52.026 INFO:tasks.workunit.client.0.trial034.stdout: 2147483647 2026-02-22T21:54:52.026 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.026 INFO:tasks.workunit.client.0.trial034.stdout: "acting": [ 2026-02-22T21:54:52.026 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T21:54:52.026 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T21:54:52.026 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T21:54:52.026 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T21:54:52.027 INFO:tasks.workunit.client.0.trial034.stdout: 2147483647 2026-02-22T21:54:52.027 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.027 INFO:tasks.workunit.client.0.trial034.stdout: "avail_no_missing": [ 2026-02-22T21:54:52.027 INFO:tasks.workunit.client.0.trial034.stdout: "3(0)", 2026-02-22T21:54:52.027 INFO:tasks.workunit.client.0.trial034.stdout: "0(3)", 2026-02-22T21:54:52.027 INFO:tasks.workunit.client.0.trial034.stdout: "1(1)", 2026-02-22T21:54:52.027 INFO:tasks.workunit.client.0.trial034.stdout: "4(2)" 2026-02-22T21:54:52.027 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.027 INFO:tasks.workunit.client.0.trial034.stdout: "object_location_counts": [ 2026-02-22T21:54:52.027 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.027 INFO:tasks.workunit.client.0.trial034.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-22T21:54:52.028 INFO:tasks.workunit.client.0.trial034.stdout: "objects": 301 2026-02-22T21:54:52.028 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.028 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.028 INFO:tasks.workunit.client.0.trial034.stdout: "blocked_by": [], 2026-02-22T21:54:52.028 INFO:tasks.workunit.client.0.trial034.stdout: "up_primary": 3, 2026-02-22T21:54:52.028 INFO:tasks.workunit.client.0.trial034.stdout: "acting_primary": 3, 2026-02-22T21:54:52.028 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [] 2026-02-22T21:54:52.028 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.028 INFO:tasks.workunit.client.0.trial034.stdout: "empty": 0, 2026-02-22T21:54:52.028 INFO:tasks.workunit.client.0.trial034.stdout: "dne": 0, 2026-02-22T21:54:52.028 INFO:tasks.workunit.client.0.trial034.stdout: "incomplete": 0, 2026-02-22T21:54:52.029 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 56, 2026-02-22T21:54:52.029 INFO:tasks.workunit.client.0.trial034.stdout: "hit_set_history": { 2026-02-22T21:54:52.029 INFO:tasks.workunit.client.0.trial034.stdout: "current_last_update": "0'0", 2026-02-22T21:54:52.029 INFO:tasks.workunit.client.0.trial034.stdout: "history": [] 2026-02-22T21:54:52.029 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.029 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.029 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.029 INFO:tasks.workunit.client.0.trial034.stdout: "peer": "1(1)", 2026-02-22T21:54:52.029 INFO:tasks.workunit.client.0.trial034.stdout: "pgid": "2.0s1", 2026-02-22T21:54:52.029 INFO:tasks.workunit.client.0.trial034.stdout: "shared": "1", 2026-02-22T21:54:52.029 INFO:tasks.workunit.client.0.trial034.stdout: "last_update": "46'301", 2026-02-22T21:54:52.029 INFO:tasks.workunit.client.0.trial034.stdout: "last_complete": "46'301", 2026-02-22T21:54:52.030 INFO:tasks.workunit.client.0.trial034.stdout: "log_tail": "46'200", 2026-02-22T21:54:52.030 INFO:tasks.workunit.client.0.trial034.stdout: "last_user_version": 301, 2026-02-22T21:54:52.030 INFO:tasks.workunit.client.0.trial034.stdout: "last_backfill": "MAX", 2026-02-22T21:54:52.030 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [], 2026-02-22T21:54:52.030 INFO:tasks.workunit.client.0.trial034.stdout: "history": { 2026-02-22T21:54:52.030 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_created": 40, 2026-02-22T21:54:52.030 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_pool_created": 40, 2026-02-22T21:54:52.030 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 56, 2026-02-22T21:54:52.030 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_started": 55, 2026-02-22T21:54:52.030 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 41, 2026-02-22T21:54:52.030 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_clean": 40, 2026-02-22T21:54:52.030 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_split": 0, 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_marked_full": 0, 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "same_up_since": 55, 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "same_interval_since": 55, 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "same_primary_since": 40, 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "prior_readable_until_ub": 0 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "stats": { 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "version": "46'301", 2026-02-22T21:54:52.031 INFO:tasks.workunit.client.0.trial034.stdout: "reported_seq": 330, 2026-02-22T21:54:52.032 INFO:tasks.workunit.client.0.trial034.stdout: "reported_epoch": 46, 2026-02-22T21:54:52.032 INFO:tasks.workunit.client.0.trial034.stdout: "state": "active+undersized+degraded", 2026-02-22T21:54:52.032 INFO:tasks.workunit.client.0.trial034.stdout: "last_fresh": "2026-02-22T21:54:23.820398+0000", 2026-02-22T21:54:52.032 INFO:tasks.workunit.client.0.trial034.stdout: "last_change": "2026-02-22T21:54:14.890666+0000", 2026-02-22T21:54:52.032 INFO:tasks.workunit.client.0.trial034.stdout: "last_active": "2026-02-22T21:54:23.820398+0000", 2026-02-22T21:54:52.032 INFO:tasks.workunit.client.0.trial034.stdout: "last_peered": "2026-02-22T21:54:23.820398+0000", 2026-02-22T21:54:52.032 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean": "2026-02-22T21:54:14.292907+0000", 2026-02-22T21:54:52.032 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_active": "2026-02-22T21:54:14.890666+0000", 2026-02-22T21:54:52.032 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_peered": "2026-02-22T21:54:14.890666+0000", 2026-02-22T21:54:52.032 INFO:tasks.workunit.client.0.trial034.stdout: "last_unstale": "2026-02-22T21:54:23.820398+0000", 2026-02-22T21:54:52.032 INFO:tasks.workunit.client.0.trial034.stdout: "last_undegraded": "2026-02-22T21:54:14.888829+0000", 2026-02-22T21:54:52.032 INFO:tasks.workunit.client.0.trial034.stdout: "last_fullsized": "2026-02-22T21:54:14.888736+0000", 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "mapping_epoch": 55, 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "log_start": "46'200", 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_start": "46'200", 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "created": 40, 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 41, 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "parent": "0.0", 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "parent_split_bits": 0, 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.033 INFO:tasks.workunit.client.0.trial034.stdout: "objects_scrubbed": 0, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "log_size": 101, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "log_dups_size": 200, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_size": 101, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "stats_invalid": false, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "dirty_stats_invalid": false, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "omap_stats_invalid": false, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_stats_invalid": false, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_bytes_stats_invalid": false, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "pin_stats_invalid": false, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "manifest_stats_invalid": false, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrimq_len": 0, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_duration": 0, 2026-02-22T21:54:52.034 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-24T00:19:14.591533+0000", 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_duration": 0, 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "objects_trimmed": 0, 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrim_duration": 0, 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "stat_sum": { 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes": 1232896, 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects": 301, 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_clones": 0, 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_copies": 1505, 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing": 1, 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_degraded": 301, 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_misplaced": 0, 2026-02-22T21:54:52.035 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_unfound": 0, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_dirty": 301, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_whiteouts": 0, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_read": 0, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_read_kb": 0, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_write": 301, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_write_kb": 1204, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_scrub_errors": 0, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_deep_scrub_errors": 0, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_recovered": 0, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_recovered": 0, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_keys_recovered": 0, 2026-02-22T21:54:52.036 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_omap": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_kb": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_kb": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_promote": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_high": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_low": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_some": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_full": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_pinned": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_legacy_snapsets": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_large_omap_objects": 0, 2026-02-22T21:54:52.037 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_manifest": 0, 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_bytes": 0, 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_keys": 0, 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_repaired": 0 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: "up": [ 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: 2147483647 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: "acting": [ 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T21:54:52.038 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: 2147483647 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: "avail_no_missing": [ 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: "3(0)", 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: "0(3)", 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: "1(1)", 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: "4(2)" 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: "object_location_counts": [ 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: "objects": 301 2026-02-22T21:54:52.039 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: "blocked_by": [], 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: "up_primary": 3, 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: "acting_primary": 3, 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [] 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: "empty": 0, 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: "dne": 0, 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: "incomplete": 0, 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 56, 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: "hit_set_history": { 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: "current_last_update": "0'0", 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: "history": [] 2026-02-22T21:54:52.040 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "peer": "4(2)", 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "pgid": "2.0s2", 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "shared": "2", 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "last_update": "46'301", 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "last_complete": "46'301", 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "log_tail": "46'200", 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "last_user_version": 301, 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "last_backfill": "MAX", 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [], 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "history": { 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_created": 40, 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_pool_created": 40, 2026-02-22T21:54:52.041 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 56, 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_started": 55, 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 41, 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_clean": 40, 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_split": 0, 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_marked_full": 0, 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "same_up_since": 55, 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "same_interval_since": 55, 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "same_primary_since": 40, 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "prior_readable_until_ub": 0 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "stats": { 2026-02-22T21:54:52.042 INFO:tasks.workunit.client.0.trial034.stdout: "version": "46'301", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "reported_seq": 330, 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "reported_epoch": 46, 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "state": "active+undersized+degraded", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "last_fresh": "2026-02-22T21:54:23.820398+0000", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "last_change": "2026-02-22T21:54:14.890666+0000", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "last_active": "2026-02-22T21:54:23.820398+0000", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "last_peered": "2026-02-22T21:54:23.820398+0000", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean": "2026-02-22T21:54:14.292907+0000", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_active": "2026-02-22T21:54:14.890666+0000", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_peered": "2026-02-22T21:54:14.890666+0000", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "last_unstale": "2026-02-22T21:54:23.820398+0000", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "last_undegraded": "2026-02-22T21:54:14.888829+0000", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "last_fullsized": "2026-02-22T21:54:14.888736+0000", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "mapping_epoch": 55, 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "log_start": "46'200", 2026-02-22T21:54:52.043 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_start": "46'200", 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "created": 40, 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 41, 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "parent": "0.0", 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "parent_split_bits": 0, 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T21:54:08.162270+0000", 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "objects_scrubbed": 0, 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "log_size": 101, 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "log_dups_size": 200, 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_size": 101, 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "stats_invalid": false, 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "dirty_stats_invalid": false, 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "omap_stats_invalid": false, 2026-02-22T21:54:52.044 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_stats_invalid": false, 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_bytes_stats_invalid": false, 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "pin_stats_invalid": false, 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "manifest_stats_invalid": false, 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrimq_len": 0, 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_duration": 0, 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-24T00:19:14.591533+0000", 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_duration": 0, 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "objects_trimmed": 0, 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrim_duration": 0, 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "stat_sum": { 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes": 1232896, 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects": 301, 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_clones": 0, 2026-02-22T21:54:52.045 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_copies": 1505, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_degraded": 301, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_misplaced": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_unfound": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_dirty": 301, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_whiteouts": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_read": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_read_kb": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_write": 301, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_write_kb": 1204, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_scrub_errors": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_deep_scrub_errors": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_recovered": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_recovered": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_keys_recovered": 0, 2026-02-22T21:54:52.046 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_omap": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_kb": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_kb": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_promote": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_high": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_low": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_some": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_full": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_pinned": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_legacy_snapsets": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_large_omap_objects": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_manifest": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_bytes": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_keys": 0, 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_repaired": 0 2026-02-22T21:54:52.047 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: "up": [ 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: 2147483647 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: "acting": [ 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: 2147483647 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: "avail_no_missing": [ 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: "3(0)", 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: "0(3)", 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: "1(1)", 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: "4(2)" 2026-02-22T21:54:52.048 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "object_location_counts": [ 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "objects": 301 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "blocked_by": [], 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "up_primary": 3, 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "acting_primary": 3, 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [] 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "empty": 0, 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "dne": 0, 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "incomplete": 0, 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 56, 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "hit_set_history": { 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "current_last_update": "0'0", 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: "history": [] 2026-02-22T21:54:52.049 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: "recovery_state": [ 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: "name": "Started/Primary/Active", 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: "enter_time": "2026-02-22T21:54:45.525225+0000", 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: "might_have_unfound": [ 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: "osd": "0(3)", 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: "status": "already probed" 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: "osd": "1(1)", 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: "status": "already probed" 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: "osd": "2(4)", 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: "status": "osd is down" 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.050 INFO:tasks.workunit.client.0.trial034.stdout: "osd": "4(2)", 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "status": "already probed" 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "recovery_progress": { 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "backfill_targets": [], 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "waiting_on_backfill": [], 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "last_backfill_started": "MIN", 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "backfill_info": { 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "begin": "MIN", 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "end": "MIN", 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "objects": [] 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "peer_backfill_info": [], 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "backfills_in_flight": [], 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "recovering": [], 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "pg_backend": { 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "recovery_ops": [], 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: "read_ops": [] 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.051 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: "name": "Started", 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: "enter_time": "2026-02-22T21:54:45.425331+0000" 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: "scrubber": { 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: "active": false, 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: "must_scrub": false, 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: "must_deep_scrub": false, 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: "must_repair": false, 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: "need_auto": false, 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_reg_stamp": "2026-02-24T02:31:43.985695+0000", 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: "schedule": "no scrub is scheduled" 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout: "agent_state": {} 2026-02-22T21:54:52.052 INFO:tasks.workunit.client.0.trial034.stdout:} 2026-02-22T21:54:52.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-22T21:54:52.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: grep -q obj250 2026-02-22T21:54:52.090 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-22T21:54:52.090 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2026-02-22T21:54:52.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: check=true 2026-02-22T21:54:52.167 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: test true == true 2026-02-22T21:54:52.167 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-22T21:54:52.167 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2026-02-22T21:54:52.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2026-02-22T21:54:52.242 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: check='osd is down' 2026-02-22T21:54:52.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2026-02-22T21:54:52.242 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2026-02-22T21:54:52.242 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2026-02-22T21:54:52.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2026-02-22T21:54:52.320 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: check='2(4)' 2026-02-22T21:54:52.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2026-02-22T21:54:52.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:581: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2026-02-22T21:54:52.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T21:54:52.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T21:54:52.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T21:54:52.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T21:54:52.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T21:54:52.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:54:52.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:54:52.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:54:52.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:54:52.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T21:54:52.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T21:54:52.322 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:54:52.322 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T21:54:52.322 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:54:52.322 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T21:54:52.322 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:54:52.322 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:54:52.322 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:54:52.322 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:54:52.322 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:54:52.322 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:54:52.322 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:54:52.322 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:54:52.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:54:52.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:54:52.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:54:52.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:54:52.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:54:52.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:54:52.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T21:54:52.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T21:54:52.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T21:54:52.324 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T21:54:52.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T21:54:52.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:54:52.325 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-22T21:54:52.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T21:54:52.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T21:54:52.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T21:54:52.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T21:54:52.335 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:52.335+0000 7f96e80298c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:54:52.336 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:52.335+0000 7f96e80298c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:54:52.337 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:52.335+0000 7f96e80298c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:54:52.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T21:54:52.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:54:52.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T21:54:52.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:54:52.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:54:52.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:52.516 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:54:52.516 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:54:52.516 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:52.516 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:54:52.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:54:53.612 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:53.611+0000 7f96e80298c0 -1 Falling back to public interface 2026-02-22T21:54:53.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:54:53.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:53.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:54:53.714 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:54:53.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:53.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:54:53.906 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:54:54.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:54:54.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:54.907 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:54:54.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:54:54.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:54.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:54:55.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:54:56.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:54:56.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:56.101 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:54:56.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:54:56.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:56.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:54:56.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:54:56.869 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:54:56.867+0000 7f96e80298c0 -1 osd.2 54 log_to_monitors true 2026-02-22T21:54:57.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:54:57.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:54:57.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:54:57.296 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:54:57.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:54:57.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:54:57.488 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 58 up_thru 20 down_at 55 last_clean_interval [48,54) [v2:127.0.0.1:6818/1213330355,v1:127.0.0.1:6819/1213330355] [v2:127.0.0.1:6820/1213330355,v1:127.0.0.1:6821/1213330355] exists,up 44dbedf4-066d-448d-bd05-d807a608b692 2026-02-22T21:54:57.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:54:57.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:54:57.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:54:57.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2026-02-22T21:55:02.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:585: TEST_ec_backfill_unfound: test 124 = 124 2026-02-22T21:55:02.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2026-02-22T21:55:02.561 INFO:tasks.workunit.client.0.trial034.stderr:pg has no unfound objects 2026-02-22T21:55:02.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:589: TEST_ec_backfill_unfound: wait_for_clean 2026-02-22T21:55:02.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:55:02.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:55:02.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:55:02.570 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:55:02.570 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:55:02.570 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:55:02.571 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:55:02.571 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:55:02.571 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:55:02.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:55:02.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:55:02.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:55:02.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:55:02.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:55:02.680 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:55:02.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:55:02.874 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:55:02.874 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:55:02.874 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:55:02.874 INFO:tasks.workunit.client.0.trial034.stderr:4' 2026-02-22T21:55:02.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:55:02.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:55:02.874 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:55:02.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803799 2026-02-22T21:55:02.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803799 2026-02-22T21:55:02.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803799' 2026-02-22T21:55:02.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:55:02.949 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:55:03.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607573 2026-02-22T21:55:03.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607573 2026-02-22T21:55:03.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803799 1-51539607573' 2026-02-22T21:55:03.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:55:03.031 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:55:03.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=249108103172 2026-02-22T21:55:03.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 249108103172 2026-02-22T21:55:03.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803799 1-51539607573 2-249108103172' 2026-02-22T21:55:03.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:55:03.106 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:55:03.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=115964117009 2026-02-22T21:55:03.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 115964117009 2026-02-22T21:55:03.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803799 1-51539607573 2-249108103172 3-115964117009' 2026-02-22T21:55:03.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:55:03.182 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:55:03.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=150323855376 2026-02-22T21:55:03.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 150323855376 2026-02-22T21:55:03.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803799 1-51539607573 2-249108103172 3-115964117009 4-150323855376' 2026-02-22T21:55:03.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:55:03.257 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803799 2026-02-22T21:55:03.257 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:55:03.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:55:03.258 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803799 2026-02-22T21:55:03.258 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:55:03.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803799 2026-02-22T21:55:03.260 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803799 2026-02-22T21:55:03.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803799' 2026-02-22T21:55:03.260 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:55:03.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803797 -lt 25769803799 2026-02-22T21:55:03.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:55:04.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:55:04.469 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:55:04.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803799 -lt 25769803799 2026-02-22T21:55:04.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:55:04.665 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607573 2026-02-22T21:55:04.665 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:55:04.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:55:04.667 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607573 2026-02-22T21:55:04.667 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:55:04.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607573 2026-02-22T21:55:04.669 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607573 2026-02-22T21:55:04.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607573' 2026-02-22T21:55:04.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:55:04.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607573 -lt 51539607573 2026-02-22T21:55:04.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:55:04.874 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-249108103172 2026-02-22T21:55:04.874 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:55:04.875 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:55:04.876 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-249108103172 2026-02-22T21:55:04.876 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:55:04.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=249108103172 2026-02-22T21:55:04.877 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 249108103172 2026-02-22T21:55:04.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 249108103172' 2026-02-22T21:55:04.878 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:55:05.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 249108103172 -lt 249108103172 2026-02-22T21:55:05.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:55:05.075 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-115964117009 2026-02-22T21:55:05.075 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:55:05.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:55:05.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-115964117009 2026-02-22T21:55:05.076 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:55:05.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=115964117009 2026-02-22T21:55:05.078 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 115964117009 2026-02-22T21:55:05.078 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 115964117009' 2026-02-22T21:55:05.078 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:55:05.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 115964117009 -lt 115964117009 2026-02-22T21:55:05.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:55:05.282 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-150323855376 2026-02-22T21:55:05.282 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:55:05.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:55:05.284 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-150323855376 2026-02-22T21:55:05.284 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:55:05.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=150323855376 2026-02-22T21:55:05.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 150323855376' 2026-02-22T21:55:05.285 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 150323855376 2026-02-22T21:55:05.285 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:55:05.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 150323855376 -lt 150323855376 2026-02-22T21:55:05.489 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:55:05.489 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:55:05.490 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:55:05.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T21:55:05.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:55:05.750 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:55:05.750 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:55:05.750 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:55:05.750 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:55:05.750 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:55:05.750 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:55:05.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T21:55:05.947 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:55:05.948 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:55:05.948 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:55:06.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T21:55:06.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:55:06.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:55:06.211 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: seq 1 300 2026-02-22T21:55:06.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2026-02-22T21:55:06.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2026-02-22T21:55:06.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2026-02-22T21:55:06.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2026-02-22T21:55:06.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2026-02-22T21:55:06.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.322 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.322 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2026-02-22T21:55:06.322 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2026-02-22T21:55:06.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2026-02-22T21:55:06.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2026-02-22T21:55:06.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.414 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.414 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2026-02-22T21:55:06.414 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2026-02-22T21:55:06.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2026-02-22T21:55:06.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.482 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.482 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2026-02-22T21:55:06.482 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2026-02-22T21:55:06.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.522 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.522 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2026-02-22T21:55:06.522 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2026-02-22T21:55:06.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2026-02-22T21:55:06.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2026-02-22T21:55:06.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2026-02-22T21:55:06.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2026-02-22T21:55:06.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2026-02-22T21:55:06.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2026-02-22T21:55:06.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2026-02-22T21:55:06.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2026-02-22T21:55:06.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2026-02-22T21:55:06.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.772 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2026-02-22T21:55:06.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2026-02-22T21:55:06.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2026-02-22T21:55:06.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.834 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2026-02-22T21:55:06.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.857 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.857 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2026-02-22T21:55:06.857 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2026-02-22T21:55:06.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2026-02-22T21:55:06.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2026-02-22T21:55:06.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.941 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.941 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2026-02-22T21:55:06.941 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.966 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.966 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2026-02-22T21:55:06.966 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2026-02-22T21:55:06.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:06.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:06.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2026-02-22T21:55:06.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2026-02-22T21:55:07.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2026-02-22T21:55:07.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2026-02-22T21:55:07.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2026-02-22T21:55:07.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2026-02-22T21:55:07.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.122 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.122 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2026-02-22T21:55:07.122 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2026-02-22T21:55:07.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.171 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.172 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.172 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2026-02-22T21:55:07.172 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2026-02-22T21:55:07.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2026-02-22T21:55:07.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.243 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2026-02-22T21:55:07.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2026-02-22T21:55:07.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2026-02-22T21:55:07.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2026-02-22T21:55:07.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.329 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.330 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.330 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2026-02-22T21:55:07.331 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2026-02-22T21:55:07.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2026-02-22T21:55:07.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.389 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2026-02-22T21:55:07.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2026-02-22T21:55:07.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2026-02-22T21:55:07.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2026-02-22T21:55:07.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2026-02-22T21:55:07.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2026-02-22T21:55:07.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2026-02-22T21:55:07.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2026-02-22T21:55:07.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2026-02-22T21:55:07.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2026-02-22T21:55:07.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2026-02-22T21:55:07.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2026-02-22T21:55:07.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2026-02-22T21:55:07.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2026-02-22T21:55:07.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2026-02-22T21:55:07.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.732 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.734 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.734 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2026-02-22T21:55:07.734 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2026-02-22T21:55:07.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2026-02-22T21:55:07.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2026-02-22T21:55:07.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2026-02-22T21:55:07.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.834 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2026-02-22T21:55:07.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.860 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2026-02-22T21:55:07.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2026-02-22T21:55:07.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2026-02-22T21:55:07.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2026-02-22T21:55:07.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.938 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2026-02-22T21:55:07.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.961 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2026-02-22T21:55:07.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2026-02-22T21:55:07.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:07.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:07.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2026-02-22T21:55:07.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2026-02-22T21:55:08.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2026-02-22T21:55:08.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2026-02-22T21:55:08.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2026-02-22T21:55:08.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2026-02-22T21:55:08.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.111 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.112 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.112 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2026-02-22T21:55:08.112 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2026-02-22T21:55:08.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.153 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.154 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.154 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2026-02-22T21:55:08.154 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.171 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.172 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.173 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2026-02-22T21:55:08.173 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2026-02-22T21:55:08.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2026-02-22T21:55:08.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2026-02-22T21:55:08.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2026-02-22T21:55:08.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2026-02-22T21:55:08.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2026-02-22T21:55:08.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.326 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2026-02-22T21:55:08.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2026-02-22T21:55:08.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2026-02-22T21:55:08.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.383 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.384 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.384 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2026-02-22T21:55:08.384 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2026-02-22T21:55:08.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2026-02-22T21:55:08.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2026-02-22T21:55:08.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2026-02-22T21:55:08.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2026-02-22T21:55:08.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2026-02-22T21:55:08.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2026-02-22T21:55:08.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2026-02-22T21:55:08.568 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2026-02-22T21:55:08.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2026-02-22T21:55:08.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2026-02-22T21:55:08.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2026-02-22T21:55:08.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2026-02-22T21:55:08.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2026-02-22T21:55:08.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2026-02-22T21:55:08.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.763 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2026-02-22T21:55:08.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2026-02-22T21:55:08.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2026-02-22T21:55:08.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2026-02-22T21:55:08.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.854 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2026-02-22T21:55:08.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.875 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.875 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2026-02-22T21:55:08.875 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2026-02-22T21:55:08.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2026-02-22T21:55:08.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2026-02-22T21:55:08.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2026-02-22T21:55:08.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2026-02-22T21:55:08.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:08.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:08.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2026-02-22T21:55:08.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2026-02-22T21:55:09.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.042 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.042 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2026-02-22T21:55:09.042 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2026-02-22T21:55:09.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2026-02-22T21:55:09.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.111 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.112 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.113 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2026-02-22T21:55:09.113 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2026-02-22T21:55:09.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.159 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2026-02-22T21:55:09.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.184 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.184 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2026-02-22T21:55:09.184 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2026-02-22T21:55:09.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2026-02-22T21:55:09.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2026-02-22T21:55:09.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2026-02-22T21:55:09.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2026-02-22T21:55:09.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2026-02-22T21:55:09.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2026-02-22T21:55:09.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.365 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.365 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2026-02-22T21:55:09.365 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2026-02-22T21:55:09.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2026-02-22T21:55:09.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2026-02-22T21:55:09.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2026-02-22T21:55:09.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2026-02-22T21:55:09.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.492 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2026-02-22T21:55:09.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2026-02-22T21:55:09.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2026-02-22T21:55:09.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2026-02-22T21:55:09.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2026-02-22T21:55:09.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2026-02-22T21:55:09.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2026-02-22T21:55:09.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2026-02-22T21:55:09.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2026-02-22T21:55:09.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2026-02-22T21:55:09.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2026-02-22T21:55:09.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2026-02-22T21:55:09.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2026-02-22T21:55:09.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2026-02-22T21:55:09.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2026-02-22T21:55:09.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2026-02-22T21:55:09.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.841 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2026-02-22T21:55:09.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2026-02-22T21:55:09.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2026-02-22T21:55:09.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2026-02-22T21:55:09.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2026-02-22T21:55:09.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2026-02-22T21:55:09.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:09.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:09.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2026-02-22T21:55:09.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2026-02-22T21:55:09.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.000 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.000 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2026-02-22T21:55:10.000 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2026-02-22T21:55:10.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2026-02-22T21:55:10.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.060 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2026-02-22T21:55:10.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.084 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2026-02-22T21:55:10.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.108 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2026-02-22T21:55:10.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.128 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.129 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.129 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2026-02-22T21:55:10.130 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.151 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.152 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.152 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2026-02-22T21:55:10.152 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2026-02-22T21:55:10.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2026-02-22T21:55:10.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2026-02-22T21:55:10.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2026-02-22T21:55:10.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2026-02-22T21:55:10.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2026-02-22T21:55:10.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2026-02-22T21:55:10.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2026-02-22T21:55:10.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2026-02-22T21:55:10.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2026-02-22T21:55:10.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2026-02-22T21:55:10.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2026-02-22T21:55:10.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2026-02-22T21:55:10.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2026-02-22T21:55:10.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2026-02-22T21:55:10.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2026-02-22T21:55:10.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.533 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.533 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2026-02-22T21:55:10.533 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2026-02-22T21:55:10.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2026-02-22T21:55:10.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2026-02-22T21:55:10.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2026-02-22T21:55:10.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2026-02-22T21:55:10.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2026-02-22T21:55:10.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2026-02-22T21:55:10.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.725 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.725 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2026-02-22T21:55:10.725 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2026-02-22T21:55:10.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2026-02-22T21:55:10.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2026-02-22T21:55:10.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2026-02-22T21:55:10.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.840 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2026-02-22T21:55:10.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2026-02-22T21:55:10.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2026-02-22T21:55:10.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2026-02-22T21:55:10.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2026-02-22T21:55:10.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.953 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.953 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2026-02-22T21:55:10.953 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2026-02-22T21:55:10.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2026-02-22T21:55:10.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:10.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:10.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2026-02-22T21:55:10.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2026-02-22T21:55:11.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2026-02-22T21:55:11.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.057 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2026-02-22T21:55:11.059 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.078 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.078 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2026-02-22T21:55:11.079 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2026-02-22T21:55:11.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.122 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2026-02-22T21:55:11.122 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.143 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2026-02-22T21:55:11.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2026-02-22T21:55:11.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.185 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2026-02-22T21:55:11.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2026-02-22T21:55:11.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2026-02-22T21:55:11.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2026-02-22T21:55:11.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2026-02-22T21:55:11.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2026-02-22T21:55:11.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2026-02-22T21:55:11.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2026-02-22T21:55:11.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2026-02-22T21:55:11.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2026-02-22T21:55:11.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2026-02-22T21:55:11.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2026-02-22T21:55:11.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.447 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.447 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2026-02-22T21:55:11.447 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2026-02-22T21:55:11.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2026-02-22T21:55:11.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2026-02-22T21:55:11.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2026-02-22T21:55:11.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.561 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.561 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2026-02-22T21:55:11.562 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2026-02-22T21:55:11.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2026-02-22T21:55:11.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2026-02-22T21:55:11.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2026-02-22T21:55:11.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2026-02-22T21:55:11.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2026-02-22T21:55:11.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.724 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.725 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2026-02-22T21:55:11.725 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2026-02-22T21:55:11.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.762 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.763 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2026-02-22T21:55:11.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.782 INFO:tasks.workunit.client.0.trial034.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2026-02-22T21:55:11.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2026-02-22T21:55:11.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2026-02-22T21:55:11.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2026-02-22T21:55:11.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.858 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.859 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.859 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2026-02-22T21:55:11.859 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2026-02-22T21:55:11.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2026-02-22T21:55:11.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2026-02-22T21:55:11.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2026-02-22T21:55:11.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2026-02-22T21:55:11.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2026-02-22T21:55:11.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2026-02-22T21:55:11.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2026-02-22T21:55:12.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2026-02-22T21:55:12.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.057 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2026-02-22T21:55:12.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2026-02-22T21:55:12.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2026-02-22T21:55:12.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.124 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2026-02-22T21:55:12.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2026-02-22T21:55:12.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.171 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.172 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.173 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2026-02-22T21:55:12.173 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2026-02-22T21:55:12.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2026-02-22T21:55:12.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2026-02-22T21:55:12.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2026-02-22T21:55:12.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2026-02-22T21:55:12.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2026-02-22T21:55:12.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2026-02-22T21:55:12.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2026-02-22T21:55:12.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2026-02-22T21:55:12.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2026-02-22T21:55:12.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2026-02-22T21:55:12.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2026-02-22T21:55:12.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2026-02-22T21:55:12.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2026-02-22T21:55:12.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.492 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.492 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2026-02-22T21:55:12.492 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2026-02-22T21:55:12.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.533 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.534 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2026-02-22T21:55:12.534 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2026-02-22T21:55:12.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2026-02-22T21:55:12.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2026-02-22T21:55:12.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.626 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.626 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2026-02-22T21:55:12.626 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2026-02-22T21:55:12.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2026-02-22T21:55:12.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2026-02-22T21:55:12.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2026-02-22T21:55:12.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2026-02-22T21:55:12.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2026-02-22T21:55:12.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2026-02-22T21:55:12.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2026-02-22T21:55:12.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2026-02-22T21:55:12.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2026-02-22T21:55:12.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2026-02-22T21:55:12.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2026-02-22T21:55:12.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2026-02-22T21:55:12.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:602: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T21:55:12.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2026-02-22T21:55:12.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T21:55:12.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T21:55:13.144 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T21:55:13.153 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T21:55:13.415 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T21:55:13.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T21:55:13.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T21:55:13.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:55:13.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T21:55:13.425 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:55:13.425 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:55:13.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:55:13.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:55:13.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:55:13.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:55:13.738 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:55:13.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:55:13.740 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:55:13.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:55:13.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:55:13.742 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:55:13.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:55:13.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:55:13.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:55:13.744 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:55:13.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:55:13.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:55:13.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T21:55:13.789 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:55:13.789 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:13.789 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:13.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T21:55:13.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:55:13.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:55:13.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T21:55:13.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T21:55:13.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T21:55:13.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T21:55:13.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T21:55:13.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:55:13.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T21:55:13.792 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:55:13.792 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:55:13.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:55:13.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:55:13.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:55:13.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:55:13.794 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:55:13.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:55:13.795 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:55:13.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:55:13.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:55:13.796 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:55:13.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:55:13.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:55:13.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:55:13.798 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:55:13.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:55:13.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:55:13.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T21:55:13.802 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:55:13.802 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:13.802 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:13.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T21:55:13.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:55:13.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:55:13.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T21:55:13.804 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:55:13.804 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:13.804 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:13.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T21:55:13.806 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:55:13.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:55:13.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:55:13.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T21:55:13.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T21:55:13.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T21:55:13.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:55:13.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:55:13.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:55:13.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T21:55:13.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T21:55:13.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:55:13.829 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:55:13.829 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:55:13.829 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:55:13.829 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:13.829 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:13.829 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:55:13.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T21:55:13.857 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:55:13.858 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:55:13.858 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:55:13.858 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:55:13.858 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:55:13.860 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:55:13.860 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:55:13.860 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:55:13.860 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:55:13.860 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:55:13.860 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:13.860 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:13.860 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:55:13.860 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:55:13.861 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T21:55:13.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:55:13.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:55:13.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:55:13.936 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:55:13.936 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:55:13.937 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:55:13.937 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:55:13.937 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:55:13.937 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:55:13.937 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:13.937 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:13.937 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:55:13.938 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:55:13.938 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T21:55:13.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T21:55:13.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T21:55:13.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:55:13.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:55:13.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:55:13.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T21:55:13.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:55:14.106 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:55:14.106 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:55:14.106 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:55:14.106 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:55:14.106 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:14.106 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:14.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:55:14.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:55:14.108 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:55:14.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T21:55:14.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T21:55:14.239 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T21:55:14.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:55:15.248 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T21:55:15.248 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:55:15.248 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:55:15.248 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:55:15.249 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:15.249 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:15.249 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:55:15.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T21:55:15.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T21:55:15.298 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:55:15.304 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T21:55:15.305 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:55:13.847+0000 7feafa4ecec0 0 load: jerasure load: lrc 2026-02-22T21:55:15.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2026-02-22T21:55:15.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:359: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2026-02-22T21:55:15.307 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_ec_object_attr_read_error: local objname=myobject 2026-02-22T21:55:15.307 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:362: TEST_ec_object_attr_read_error: setup_osds 7 2026-02-22T21:55:15.307 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2026-02-22T21:55:15.307 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T21:55:15.307 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2026-02-22T21:55:15.307 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2026-02-22T21:55:15.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:55:15.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T21:55:15.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:55:15.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:55:15.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:55:15.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:55:15.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T21:55:15.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:55:15.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:55:15.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:55:15.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:55:15.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T21:55:15.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T21:55:15.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:55:15.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:55:15.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:55:15.312 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:55:15.313 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:55:15.313 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:55:15.313 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:55:15.313 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:15.313 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:15.313 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:55:15.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:55:15.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:55:15.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:55:15.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:55:15.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:55:15.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:55:15.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:55:15.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:55:15.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:55:15.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:55:15.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:55:15.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:55:15.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T21:55:15.314 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:55:15.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c4e88e4-c81a-4635-b458-6bfafa51d064 2026-02-22T21:55:15.315 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 9c4e88e4-c81a-4635-b458-6bfafa51d064 2026-02-22T21:55:15.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9c4e88e4-c81a-4635-b458-6bfafa51d064' 2026-02-22T21:55:15.316 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:55:15.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDe5tpxh5dExAAqtl6XXUSoXLE9iFUtU4oiQ== 2026-02-22T21:55:15.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDe5tpxh5dExAAqtl6XXUSoXLE9iFUtU4oiQ=="}' 2026-02-22T21:55:15.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c4e88e4-c81a-4635-b458-6bfafa51d064 -i td/test-erasure-eio/0/new.json 2026-02-22T21:55:15.433 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:55:15.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T21:55:15.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDe5tpxh5dExAAqtl6XXUSoXLE9iFUtU4oiQ== --osd-uuid 9c4e88e4-c81a-4635-b458-6bfafa51d064 2026-02-22T21:55:15.458 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:15.459+0000 7fe6bad928c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:15.460 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:15.459+0000 7fe6bad928c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:15.461 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:15.459+0000 7fe6bad928c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:15.461 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:15.459+0000 7fe6bad928c0 -1 bdev(0x56433dfd3c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:55:15.461 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:15.459+0000 7fe6bad928c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T21:55:18.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T21:55:18.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:55:18.016 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:55:18.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:55:18.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:55:18.285 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:55:18.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:55:18.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:55:18.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:55:18.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:55:18.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:55:18.309 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:18.307+0000 7fb791d828c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:18.310 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:18.311+0000 7fb791d828c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:18.311 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:18.311+0000 7fb791d828c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:18.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:55:18.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:55:18.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:55:18.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:55:18.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:55:18.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:18.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:55:18.489 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:55:18.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:18.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:55:18.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:19.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:19.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:19.683 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:55:19.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:55:19.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:19.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:55:19.832 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:19.831+0000 7fb791d828c0 -1 Falling back to public interface 2026-02-22T21:55:19.875 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:20.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:20.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:20.877 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:55:20.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:55:20.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:20.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:55:21.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:22.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:22.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:22.068 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:55:22.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:55:22.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:22.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:55:22.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:22.858 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:22.855+0000 7fb791d828c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:55:23.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:23.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:23.263 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:55:23.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:55:23.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:23.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:55:23.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:24.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:24.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:24.463 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:55:24.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:55:24.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:24.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:55:24.657 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1483560834,v1:127.0.0.1:6803/1483560834] [v2:127.0.0.1:6804/1483560834,v1:127.0.0.1:6805/1483560834] exists,up 9c4e88e4-c81a-4635-b458-6bfafa51d064 2026-02-22T21:55:24.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:55:24.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:55:24.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:55:24.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:55:24.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T21:55:24.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:55:24.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:55:24.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T21:55:24.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:55:24.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T21:55:24.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:55:24.662 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:55:24.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:55:24.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:55:24.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T21:55:24.664 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:55:24.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b84d0f4c-b39f-444f-85fc-e0c6bd865d63 2026-02-22T21:55:24.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b84d0f4c-b39f-444f-85fc-e0c6bd865d63' 2026-02-22T21:55:24.665 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 b84d0f4c-b39f-444f-85fc-e0c6bd865d63 2026-02-22T21:55:24.665 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:55:24.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBMe5tpZngzKBAANYQtKuAyb8lQH1Rqk63YAQ== 2026-02-22T21:55:24.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBMe5tpZngzKBAANYQtKuAyb8lQH1Rqk63YAQ=="}' 2026-02-22T21:55:24.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b84d0f4c-b39f-444f-85fc-e0c6bd865d63 -i td/test-erasure-eio/1/new.json 2026-02-22T21:55:24.878 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:55:24.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T21:55:24.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMe5tpZngzKBAANYQtKuAyb8lQH1Rqk63YAQ== --osd-uuid b84d0f4c-b39f-444f-85fc-e0c6bd865d63 2026-02-22T21:55:24.903 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:24.903+0000 7fd3cd0a48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:24.904 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:24.903+0000 7fd3cd0a48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:24.906 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:24.903+0000 7fd3cd0a48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:24.906 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:24.907+0000 7fd3cd0a48c0 -1 bdev(0x555a95a35c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:55:24.906 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:24.907+0000 7fd3cd0a48c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T21:55:27.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T21:55:27.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:55:27.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:55:27.200 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T21:55:27.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:55:27.465 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T21:55:27.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T21:55:27.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:55:27.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:55:27.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:55:27.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:55:27.487 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:27.487+0000 7f239ce0b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:27.489 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:27.487+0000 7f239ce0b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:27.490 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:27.491+0000 7f239ce0b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:27.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T21:55:27.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:55:27.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T21:55:27.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:55:27.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:55:27.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:27.665 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:55:27.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:55:27.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:27.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:55:27.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:28.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:28.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:28.865 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:55:28.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:55:28.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:28.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:55:28.991 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:28.991+0000 7f239ce0b8c0 -1 Falling back to public interface 2026-02-22T21:55:29.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:30.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:30.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:30.064 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:55:30.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:55:30.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:30.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:55:30.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:31.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:31.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:31.258 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:55:31.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:55:31.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:31.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:55:31.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:32.214 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:32.214+0000 7f239ce0b8c0 -1 osd.1 0 log_to_monitors true 2026-02-22T21:55:32.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:32.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:32.455 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:55:32.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:55:32.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:55:32.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:32.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:33.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:33.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:33.650 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:55:33.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:55:33.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:33.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:55:33.847 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3034373976,v1:127.0.0.1:6811/3034373976] [v2:127.0.0.1:6812/3034373976,v1:127.0.0.1:6813/3034373976] exists,up b84d0f4c-b39f-444f-85fc-e0c6bd865d63 2026-02-22T21:55:33.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:55:33.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:55:33.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:55:33.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:55:33.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T21:55:33.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:55:33.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:55:33.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T21:55:33.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:55:33.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T21:55:33.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:33.852 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b0ce6e7d-1e66-499a-8b4f-48dbfc3101eb 2026-02-22T21:55:33.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b0ce6e7d-1e66-499a-8b4f-48dbfc3101eb' 2026-02-22T21:55:33.854 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 b0ce6e7d-1e66-499a-8b4f-48dbfc3101eb 2026-02-22T21:55:33.854 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:55:33.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVe5tpZ6qQMxAAQqPhdtQM9zzIVwUd4vShAg== 2026-02-22T21:55:33.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVe5tpZ6qQMxAAQqPhdtQM9zzIVwUd4vShAg=="}' 2026-02-22T21:55:33.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b0ce6e7d-1e66-499a-8b4f-48dbfc3101eb -i td/test-erasure-eio/2/new.json 2026-02-22T21:55:34.076 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:55:34.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T21:55:34.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVe5tpZ6qQMxAAQqPhdtQM9zzIVwUd4vShAg== --osd-uuid b0ce6e7d-1e66-499a-8b4f-48dbfc3101eb 2026-02-22T21:55:34.102 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:34.102+0000 7fe276b848c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:34.103 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:34.102+0000 7fe276b848c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:34.104 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:34.102+0000 7fe276b848c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:34.104 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:34.102+0000 7fe276b848c0 -1 bdev(0x5563ec097c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:55:34.104 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:34.102+0000 7fe276b848c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T21:55:36.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T21:55:36.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:55:36.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:55:36.411 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T21:55:36.411 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:55:36.676 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T21:55:36.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T21:55:36.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:55:36.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:55:36.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:55:36.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:55:36.698 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:36.698+0000 7f71b67448c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:36.700 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:36.698+0000 7f71b67448c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:36.701 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:36.698+0000 7f71b67448c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:36.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T21:55:36.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:55:36.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T21:55:36.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:55:36.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:55:36.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:36.877 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:55:36.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:55:36.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:36.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:55:37.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:38.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:38.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:38.075 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:55:38.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:55:38.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:38.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:55:38.224 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:38.222+0000 7f71b67448c0 -1 Falling back to public interface 2026-02-22T21:55:38.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:39.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:39.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:39.273 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:55:39.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:55:39.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:39.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:55:39.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:40.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:40.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:40.465 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:55:40.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:55:40.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:40.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:55:40.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:41.239 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:41.238+0000 7f71b67448c0 -1 osd.2 0 log_to_monitors true 2026-02-22T21:55:41.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:41.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:41.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:55:41.660 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:55:41.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:41.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:55:41.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:42.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:42.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:42.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:55:42.864 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:55:42.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:42.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:55:43.059 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1287856827,v1:127.0.0.1:6819/1287856827] [v2:127.0.0.1:6820/1287856827,v1:127.0.0.1:6821/1287856827] exists,up b0ce6e7d-1e66-499a-8b4f-48dbfc3101eb 2026-02-22T21:55:43.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:55:43.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:55:43.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:55:43.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:55:43.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T21:55:43.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:55:43.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:55:43.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T21:55:43.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:55:43.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T21:55:43.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:55:43.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:55:43.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:55:43.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:55:43.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T21:55:43.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T21:55:43.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:55:43.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:55:43.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:55:43.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:55:43.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:55:43.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:55:43.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:55:43.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:55:43.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:55:43.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:55:43.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T21:55:43.066 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:55:43.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b8e12e03-6294-4475-b82d-1c0d5e824d81 2026-02-22T21:55:43.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b8e12e03-6294-4475-b82d-1c0d5e824d81' 2026-02-22T21:55:43.067 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 b8e12e03-6294-4475-b82d-1c0d5e824d81 2026-02-22T21:55:43.067 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:55:43.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfe5tpA3yjBBAAC2BLr2B6Vqq0X3VKm9rHRw== 2026-02-22T21:55:43.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfe5tpA3yjBBAAC2BLr2B6Vqq0X3VKm9rHRw=="}' 2026-02-22T21:55:43.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b8e12e03-6294-4475-b82d-1c0d5e824d81 -i td/test-erasure-eio/3/new.json 2026-02-22T21:55:43.270 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:55:43.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T21:55:43.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfe5tpA3yjBBAAC2BLr2B6Vqq0X3VKm9rHRw== --osd-uuid b8e12e03-6294-4475-b82d-1c0d5e824d81 2026-02-22T21:55:43.295 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:43.294+0000 7fbfb41b38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:43.297 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:43.294+0000 7fbfb41b38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:43.298 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:43.298+0000 7fbfb41b38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:43.298 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:43.298+0000 7fbfb41b38c0 -1 bdev(0x55b303ac5c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T21:55:43.298 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:43.298+0000 7fbfb41b38c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T21:55:45.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T21:55:45.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:55:45.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T21:55:45.572 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T21:55:45.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:55:45.837 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T21:55:45.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T21:55:45.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:55:45.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:55:45.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:55:45.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:55:45.861 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:45.858+0000 7fef783d08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:45.863 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:45.862+0000 7fef783d08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:45.864 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:45.862+0000 7fef783d08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:46.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T21:55:46.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:55:46.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T21:55:46.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:55:46.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:55:46.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:46.034 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:55:46.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:55:46.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:46.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:55:46.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:47.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:47.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:47.237 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:55:47.238 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:55:47.238 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:47.238 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:55:47.395 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:47.394+0000 7fef783d08c0 -1 Falling back to public interface 2026-02-22T21:55:47.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:48.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:48.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:48.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:55:48.430 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:55:48.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:48.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:55:48.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:49.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:49.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:49.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:55:49.624 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:55:49.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:49.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:55:49.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:50.485 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:50.482+0000 7fef783d08c0 -1 osd.3 0 log_to_monitors true 2026-02-22T21:55:50.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:50.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:50.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:55:50.822 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:55:50.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:50.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:55:51.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:52.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:52.020 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:55:52.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:52.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:55:52.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:52.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:55:52.212 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/556025861,v1:127.0.0.1:6827/556025861] [v2:127.0.0.1:6828/556025861,v1:127.0.0.1:6829/556025861] exists,up b8e12e03-6294-4475-b82d-1c0d5e824d81 2026-02-22T21:55:52.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:55:52.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:55:52.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-22T21:55:52.216 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:55:52.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6b702f2a-cbb3-432a-988d-17c2ed5532c8 2026-02-22T21:55:52.216 INFO:tasks.workunit.client.0.trial034.stdout:add osd4 6b702f2a-cbb3-432a-988d-17c2ed5532c8 2026-02-22T21:55:52.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 6b702f2a-cbb3-432a-988d-17c2ed5532c8' 2026-02-22T21:55:52.217 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:55:52.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBoe5tpwM60DRAA1aRh238x9GpdOlFnta15eA== 2026-02-22T21:55:52.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBoe5tpwM60DRAA1aRh238x9GpdOlFnta15eA=="}' 2026-02-22T21:55:52.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6b702f2a-cbb3-432a-988d-17c2ed5532c8 -i td/test-erasure-eio/4/new.json 2026-02-22T21:55:52.432 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:55:52.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-22T21:55:52.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBoe5tpwM60DRAA1aRh238x9GpdOlFnta15eA== --osd-uuid 6b702f2a-cbb3-432a-988d-17c2ed5532c8 2026-02-22T21:55:52.458 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:52.458+0000 7fef826a28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:52.459 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:52.458+0000 7fef826a28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:52.460 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:52.458+0000 7fef826a28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:52.461 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:52.458+0000 7fef826a28c0 -1 bdev(0x558aa5a63c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-22T21:55:52.461 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:52.458+0000 7fef826a28c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-22T21:55:54.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-22T21:55:54.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:55:54.756 INFO:tasks.workunit.client.0.trial034.stdout:adding osd4 key to auth repository 2026-02-22T21:55:54.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2026-02-22T21:55:54.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:55:55.020 INFO:tasks.workunit.client.0.trial034.stdout:start osd.4 2026-02-22T21:55:55.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2026-02-22T21:55:55.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:55:55.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:55:55.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:55:55.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:55:55.042 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:55.042+0000 7f3c9ba9f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:55.044 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:55.042+0000 7f3c9ba9f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:55.045 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:55.042+0000 7f3c9ba9f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:55:55.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2026-02-22T21:55:55.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:55:55.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2026-02-22T21:55:55.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:55:55.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:55:55.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:55.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:55:55.217 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:55:55.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:55.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:55:55.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:56.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:56.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:56.409 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:55:56.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:55:56.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:56.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:55:56.539 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:56.538+0000 7f3c9ba9f8c0 -1 Falling back to public interface 2026-02-22T21:55:56.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:57.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:57.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:57.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:55:57.603 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:55:57.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:57.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:55:57.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:58.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:58.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:58.793 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:55:58.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:55:58.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:58.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:55:58.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:55:59.581 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:55:59.578+0000 7f3c9ba9f8c0 -1 osd.4 0 log_to_monitors true 2026-02-22T21:55:59.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:55:59.989 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:55:59.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:55:59.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:55:59.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:55:59.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:56:00.198 INFO:tasks.workunit.client.0.trial034.stdout:osd.4 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2752978264,v1:127.0.0.1:6835/2752978264] [v2:127.0.0.1:6836/2752978264,v1:127.0.0.1:6837/2752978264] exists,up 6b702f2a-cbb3-432a-988d-17c2ed5532c8 2026-02-22T21:56:00.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:56:00.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:56:00.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:56:00.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:56:00.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-22T21:56:00.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:56:00.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:56:00.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2026-02-22T21:56:00.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:56:00.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-22T21:56:00.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:56:00.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:56:00.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:56:00.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:56:00.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-22T21:56:00.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-22T21:56:00.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:56:00.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:56:00.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:56:00.204 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:56:00.204 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:56:00.204 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:56:00.204 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:56:00.204 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:00.204 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:00.204 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:00.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:00.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:56:00.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a00ba9a6-f3f1-493a-87b4-e7e6ecfcb1ba 2026-02-22T21:56:00.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 a00ba9a6-f3f1-493a-87b4-e7e6ecfcb1ba' 2026-02-22T21:56:00.206 INFO:tasks.workunit.client.0.trial034.stdout:add osd5 a00ba9a6-f3f1-493a-87b4-e7e6ecfcb1ba 2026-02-22T21:56:00.206 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:56:00.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwe5tp92bFDBAAGWyUKWMV7zR06yIiLDOj3w== 2026-02-22T21:56:00.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwe5tp92bFDBAAGWyUKWMV7zR06yIiLDOj3w=="}' 2026-02-22T21:56:00.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a00ba9a6-f3f1-493a-87b4-e7e6ecfcb1ba -i td/test-erasure-eio/5/new.json 2026-02-22T21:56:00.423 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:56:00.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-22T21:56:00.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwe5tp92bFDBAAGWyUKWMV7zR06yIiLDOj3w== --osd-uuid a00ba9a6-f3f1-493a-87b4-e7e6ecfcb1ba 2026-02-22T21:56:00.459 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:00.458+0000 7f7a3eaa78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:00.460 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:00.458+0000 7f7a3eaa78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:00.461 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:00.458+0000 7f7a3eaa78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:00.462 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:00.462+0000 7f7a3eaa78c0 -1 bdev(0x55a8f4337c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-22T21:56:00.462 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:00.462+0000 7f7a3eaa78c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-22T21:56:02.735 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-22T21:56:02.735 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:56:02.736 INFO:tasks.workunit.client.0.trial034.stdout:adding osd5 key to auth repository 2026-02-22T21:56:02.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2026-02-22T21:56:02.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:56:02.999 INFO:tasks.workunit.client.0.trial034.stdout:start osd.5 2026-02-22T21:56:03.000 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2026-02-22T21:56:03.001 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:56:03.001 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:56:03.001 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:56:03.001 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:56:03.022 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:03.018+0000 7f5555bd18c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:03.023 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:03.022+0000 7f5555bd18c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:03.024 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:03.022+0000 7f5555bd18c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:03.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2026-02-22T21:56:03.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:56:03.197 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:56:03.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2026-02-22T21:56:03.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:56:03.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:56:03.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:03.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:56:03.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:03.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:56:03.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:04.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:04.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:04.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:56:04.393 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:56:04.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:04.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:56:04.511 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:04.510+0000 7f5555bd18c0 -1 Falling back to public interface 2026-02-22T21:56:04.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:05.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:05.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:05.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:56:05.587 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:56:05.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:05.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:56:05.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:06.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:06.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:06.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:56:06.779 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:56:06.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:06.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:56:06.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:07.553 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:07.550+0000 7f5555bd18c0 -1 osd.5 0 log_to_monitors true 2026-02-22T21:56:07.970 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:07.970 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:56:07.970 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:07.970 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:56:07.970 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:07.970 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:56:08.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:09.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:09.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:09.166 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:56:09.167 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:56:09.167 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:09.167 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:56:09.362 INFO:tasks.workunit.client.0.trial034.stdout:osd.5 up in weight 1 up_from 42 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2279595790,v1:127.0.0.1:6843/2279595790] [v2:127.0.0.1:6844/2279595790,v1:127.0.0.1:6845/2279595790] exists,up a00ba9a6-f3f1-493a-87b4-e7e6ecfcb1ba 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:56:09.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:56:09.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-22T21:56:09.365 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:56:09.366 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e13269a3-dfcf-4e81-bedc-e5ccc8d8b297 2026-02-22T21:56:09.366 INFO:tasks.workunit.client.0.trial034.stdout:add osd6 e13269a3-dfcf-4e81-bedc-e5ccc8d8b297 2026-02-22T21:56:09.366 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 e13269a3-dfcf-4e81-bedc-e5ccc8d8b297' 2026-02-22T21:56:09.367 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:56:09.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5e5tp2s+5FhAAlHfDTD497UVLgBezNccErA== 2026-02-22T21:56:09.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5e5tp2s+5FhAAlHfDTD497UVLgBezNccErA=="}' 2026-02-22T21:56:09.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e13269a3-dfcf-4e81-bedc-e5ccc8d8b297 -i td/test-erasure-eio/6/new.json 2026-02-22T21:56:09.584 INFO:tasks.workunit.client.0.trial034.stdout:6 2026-02-22T21:56:09.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-22T21:56:09.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5e5tp2s+5FhAAlHfDTD497UVLgBezNccErA== --osd-uuid e13269a3-dfcf-4e81-bedc-e5ccc8d8b297 2026-02-22T21:56:09.611 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:09.610+0000 7f182c3158c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:09.612 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:09.610+0000 7f182c3158c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:09.613 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:09.610+0000 7f182c3158c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:09.614 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:09.610+0000 7f182c3158c0 -1 bdev(0x55ab56e31c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-22T21:56:09.614 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:09.614+0000 7f182c3158c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-22T21:56:11.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-22T21:56:11.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:56:11.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2026-02-22T21:56:11.894 INFO:tasks.workunit.client.0.trial034.stdout:adding osd6 key to auth repository 2026-02-22T21:56:11.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:56:12.159 INFO:tasks.workunit.client.0.trial034.stdout:start osd.6 2026-02-22T21:56:12.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2026-02-22T21:56:12.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:56:12.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:56:12.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:56:12.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:56:12.182 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:12.182+0000 7efeafe0d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:12.183 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:12.182+0000 7efeafe0d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:12.184 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:12.182+0000 7efeafe0d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:12.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2026-02-22T21:56:12.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:56:12.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2026-02-22T21:56:12.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:56:12.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:56:12.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:12.360 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:56:12.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:56:12.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:12.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:56:12.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:13.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:13.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:13.555 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:56:13.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:56:13.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:13.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:56:13.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:13.920 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:13.918+0000 7efeafe0d8c0 -1 Falling back to public interface 2026-02-22T21:56:14.751 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:14.751 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:14.751 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:56:14.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:56:14.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:14.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:56:14.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:15.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:15.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:15.948 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:56:15.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:56:15.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:15.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:56:16.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:16.897 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:16.894+0000 7efeafe0d8c0 -1 osd.6 0 log_to_monitors true 2026-02-22T21:56:17.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:17.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:17.148 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:56:17.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:56:17.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:17.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:56:17.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:18.389 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:18.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:18.390 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:56:18.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:56:18.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:18.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:56:18.584 INFO:tasks.workunit.client.0.trial034.stdout:osd.6 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3950667537,v1:127.0.0.1:6851/3950667537] [v2:127.0.0.1:6852/3950667537,v1:127.0.0.1:6853/3950667537] exists,up e13269a3-dfcf-4e81-bedc-e5ccc8d8b297 2026-02-22T21:56:18.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:56:18.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:56:18.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:56:18.586 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T21:56:18.586 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:56:18.586 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:56:18.586 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:56:18.586 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:18.586 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:18.586 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T21:56:18.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T21:56:18.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T21:56:18.648 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:56:18.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T21:56:18.657 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:55:19.835+0000 7fb791d828c0 0 load: jerasure load: lrc 2026-02-22T21:56:18.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2026-02-22T21:56:18.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2026-02-22T21:56:18.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T21:56:18.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T21:56:18.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-22T21:56:18.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T21:56:18.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-22T21:56:18.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T21:56:18.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-22T21:56:18.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T21:56:18.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T21:56:19.272 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T21:56:19.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:56:20.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T21:56:20.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:56:20.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:56:20.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:56:20.284 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:56:20.285 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:56:20.285 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:56:20.285 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:56:20.285 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:56:20.285 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:56:20.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:56:20.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:56:20.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:56:20.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:56:20.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:56:20.388 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:56:20.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:56:20.591 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:56:20.591 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:56:20.591 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:56:20.591 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:56:20.591 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:56:20.591 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T21:56:20.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:56:20.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:20.592 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:56:20.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803790 2026-02-22T21:56:20.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803790 2026-02-22T21:56:20.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790' 2026-02-22T21:56:20.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:20.668 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:56:20.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607564 2026-02-22T21:56:20.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607564 2026-02-22T21:56:20.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564' 2026-02-22T21:56:20.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:20.744 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:56:20.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411338 2026-02-22T21:56:20.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411338 2026-02-22T21:56:20.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338' 2026-02-22T21:56:20.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:20.822 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:56:20.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182408 2026-02-22T21:56:20.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182408 2026-02-22T21:56:20.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338 3-107374182408' 2026-02-22T21:56:20.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:20.897 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:56:20.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=146028888072 2026-02-22T21:56:20.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 146028888072 2026-02-22T21:56:20.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338 3-107374182408 4-146028888072' 2026-02-22T21:56:20.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:20.971 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:56:21.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=180388626437 2026-02-22T21:56:21.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 180388626437 2026-02-22T21:56:21.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338 3-107374182408 4-146028888072 5-180388626437' 2026-02-22T21:56:21.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:21.053 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:56:21.128 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=210453397507 2026-02-22T21:56:21.128 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 210453397507 2026-02-22T21:56:21.128 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338 3-107374182408 4-146028888072 5-180388626437 6-210453397507' 2026-02-22T21:56:21.128 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:21.129 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803790 2026-02-22T21:56:21.129 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:21.129 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:56:21.130 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803790 2026-02-22T21:56:21.130 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:21.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803790 2026-02-22T21:56:21.132 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803790 2026-02-22T21:56:21.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803790' 2026-02-22T21:56:21.132 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:56:21.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803788 -lt 25769803790 2026-02-22T21:56:21.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:56:22.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:56:22.340 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:56:22.533 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803790 -lt 25769803790 2026-02-22T21:56:22.533 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:22.534 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607564 2026-02-22T21:56:22.534 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:22.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:56:22.535 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607564 2026-02-22T21:56:22.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:22.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607564 2026-02-22T21:56:22.537 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607564 2026-02-22T21:56:22.538 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607564' 2026-02-22T21:56:22.538 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:56:22.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607564 -lt 51539607564 2026-02-22T21:56:22.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:22.730 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411338 2026-02-22T21:56:22.730 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:22.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:56:22.731 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411338 2026-02-22T21:56:22.732 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:22.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411338 2026-02-22T21:56:22.733 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 77309411338 2026-02-22T21:56:22.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411338' 2026-02-22T21:56:22.733 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:56:22.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411338 -lt 77309411338 2026-02-22T21:56:22.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:22.945 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182408 2026-02-22T21:56:22.945 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:22.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:56:22.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182408 2026-02-22T21:56:22.947 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:22.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182408 2026-02-22T21:56:22.948 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 107374182408 2026-02-22T21:56:22.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182408' 2026-02-22T21:56:22.949 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:56:23.152 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182409 -lt 107374182408 2026-02-22T21:56:23.152 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:23.153 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-146028888072 2026-02-22T21:56:23.153 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:23.154 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:56:23.155 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-146028888072 2026-02-22T21:56:23.155 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:23.156 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=146028888072 2026-02-22T21:56:23.156 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 146028888072' 2026-02-22T21:56:23.157 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 146028888072 2026-02-22T21:56:23.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:56:23.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 146028888072 -lt 146028888072 2026-02-22T21:56:23.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:23.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-180388626437 2026-02-22T21:56:23.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:23.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:56:23.365 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-180388626437 2026-02-22T21:56:23.365 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:23.366 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=180388626437 2026-02-22T21:56:23.366 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 180388626437' 2026-02-22T21:56:23.366 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 180388626437 2026-02-22T21:56:23.367 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:56:23.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 180388626437 -lt 180388626437 2026-02-22T21:56:23.561 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:23.561 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-210453397507 2026-02-22T21:56:23.561 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:23.562 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:56:23.563 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-210453397507 2026-02-22T21:56:23.563 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:23.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=210453397507 2026-02-22T21:56:23.565 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 210453397507 2026-02-22T21:56:23.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 210453397507' 2026-02-22T21:56:23.565 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:56:23.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 210453397507 -lt 210453397507 2026-02-22T21:56:23.776 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:56:23.776 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:56:23.776 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:56:24.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T21:56:24.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:56:24.035 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:56:24.035 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:56:24.035 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:56:24.035 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:56:24.035 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:56:24.035 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:56:24.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T21:56:24.236 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:56:24.237 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:56:24.237 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:56:24.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T21:56:24.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:56:24.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:56:24.501 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2026-02-22T21:56:24.501 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2026-02-22T21:56:24.501 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2026-02-22T21:56:24.501 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2026-02-22T21:56:24.501 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2026-02-22T21:56:24.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: local primary_osd=3 2026-02-22T21:56:24.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-22T21:56:24.701 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:56:24.701 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:56:24.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:56:24.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:56:24.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:56:26.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:56:26.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-22T21:56:26.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T21:56:26.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T21:56:26.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-22T21:56:26.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:56:26.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T21:56:26.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:56:26.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T21:56:26.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:56:26.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T21:56:26.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:56:26.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T21:56:26.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-22T21:56:26.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:375: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2026-02-22T21:56:26.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T21:56:26.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T21:56:26.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=mdata 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=myobject 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=1 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure myobject 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2026-02-22T21:56:26.034 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-22T21:56:26.035 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:56:26.231 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2147483647 2026-02-22T21:56:26.231 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:56:26.231 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:56:26.231 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:56:26.231 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T21:56:26.231 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2147483647 5 1 0 6 2026-02-22T21:56:26.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('2147483647' '5' '1' '0' '6') 2026-02-22T21:56:26.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T21:56:26.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=5 2026-02-22T21:56:26.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T21:56:26.232 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/5/type 2026-02-22T21:56:26.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T21:56:26.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2026-02-22T21:56:26.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T21:56:26.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=5 2026-02-22T21:56:26.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T21:56:26.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T21:56:26.234 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T21:56:26.234 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.5 2026-02-22T21:56:26.235 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2026-02-22T21:56:26.235 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2026-02-22T21:56:26.235 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:56:26.235 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:26.235 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:26.235 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.5.asok 2026-02-22T21:56:26.235 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2026-02-22T21:56:26.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T21:56:26.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T21:56:26.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T21:56:26.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.5 2026-02-22T21:56:26.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2026-02-22T21:56:26.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2026-02-22T21:56:26.298 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:56:26.298 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:26.298 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:26.298 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.5.asok 2026-02-22T21:56:26.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T21:56:26.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2026-02-22T21:56:26.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:378: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2026-02-22T21:56:26.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T21:56:26.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T21:56:26.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2026-02-22T21:56:26.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T21:56:26.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T21:56:26.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:56:26.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:56:26.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:56:26.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:56:26.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T21:56:26.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T21:56:26.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:56:26.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T21:56:26.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:56:26.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T21:56:26.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:56:26.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:56:26.363 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:56:26.363 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:26.363 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:26.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:26.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:26.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:56:26.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:56:26.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:56:26.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:56:26.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:56:26.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:56:26.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:56:26.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T21:56:26.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T21:56:26.365 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T21:56:26.365 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T21:56:26.366 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2026-02-22T21:56:26.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:56:26.367 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-22T21:56:26.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2026-02-22T21:56:26.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T21:56:26.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T21:56:26.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T21:56:26.383 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:26.382+0000 7ff2b2dd68c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:26.384 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:26.382+0000 7ff2b2dd68c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:26.385 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:26.382+0000 7ff2b2dd68c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:26.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2026-02-22T21:56:26.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:56:26.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T21:56:26.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:56:26.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:56:26.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:26.556 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:56:26.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:56:26.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:26.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:56:26.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:27.656 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:27.654+0000 7ff2b2dd68c0 -1 Falling back to public interface 2026-02-22T21:56:27.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:27.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:27.755 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:56:27.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:56:27.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:27.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:56:27.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:28.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:28.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:28.947 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:56:28.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:56:28.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:28.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:56:29.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:30.144 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:30.144 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:30.144 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:56:30.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:56:30.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:30.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:56:30.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:30.926 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:30.925+0000 7ff2b2dd68c0 -1 osd.3 57 log_to_monitors true 2026-02-22T21:56:31.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:31.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:31.347 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:56:31.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:56:31.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:31.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:56:31.538 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 61 up_thru 61 down_at 58 last_clean_interval [25,57) [v2:127.0.0.1:6826/3999946628,v1:127.0.0.1:6827/3999946628] [v2:127.0.0.1:6828/3999946628,v1:127.0.0.1:6829/3999946628] exists,up b8e12e03-6294-4475-b82d-1c0d5e824d81 2026-02-22T21:56:31.538 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:56:31.538 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:56:31.538 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:56:31.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:381: TEST_ec_object_attr_read_error: wait_for_clean 2026-02-22T21:56:31.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:56:31.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:56:31.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:56:31.539 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:56:31.539 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:56:31.539 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:56:31.539 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:56:31.540 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:56:31.540 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:56:31.646 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:56:31.646 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:56:31.646 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:56:31.646 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:56:31.646 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:56:31.646 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:56:31.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:56:31.843 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:56:31.843 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:56:31.843 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:56:31.843 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:56:31.844 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:56:31.844 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T21:56:31.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:56:31.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:31.844 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:56:31.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803794 2026-02-22T21:56:31.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803794 2026-02-22T21:56:31.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803794' 2026-02-22T21:56:31.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:31.920 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:56:31.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607568 2026-02-22T21:56:31.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607568 2026-02-22T21:56:31.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803794 1-51539607568' 2026-02-22T21:56:31.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:31.996 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:56:32.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411342 2026-02-22T21:56:32.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411342 2026-02-22T21:56:32.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803794 1-51539607568 2-77309411342' 2026-02-22T21:56:32.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:32.073 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:56:32.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=261993005059 2026-02-22T21:56:32.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 261993005059 2026-02-22T21:56:32.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803794 1-51539607568 2-77309411342 3-261993005059' 2026-02-22T21:56:32.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:32.148 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:56:32.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=146028888076 2026-02-22T21:56:32.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 146028888076 2026-02-22T21:56:32.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803794 1-51539607568 2-77309411342 3-261993005059 4-146028888076' 2026-02-22T21:56:32.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:32.224 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:56:32.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=180388626441 2026-02-22T21:56:32.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 180388626441 2026-02-22T21:56:32.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803794 1-51539607568 2-77309411342 3-261993005059 4-146028888076 5-180388626441' 2026-02-22T21:56:32.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:56:32.299 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:56:32.375 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=210453397511 2026-02-22T21:56:32.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 210453397511 2026-02-22T21:56:32.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803794 1-51539607568 2-77309411342 3-261993005059 4-146028888076 5-180388626441 6-210453397511' 2026-02-22T21:56:32.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:32.376 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803794 2026-02-22T21:56:32.377 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:32.377 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:56:32.378 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803794 2026-02-22T21:56:32.378 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:32.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803794 2026-02-22T21:56:32.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803794' 2026-02-22T21:56:32.380 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803794 2026-02-22T21:56:32.380 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:56:32.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803794 -lt 25769803794 2026-02-22T21:56:32.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:32.575 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607568 2026-02-22T21:56:32.575 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:32.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:56:32.576 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607568 2026-02-22T21:56:32.577 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:32.578 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607568 2026-02-22T21:56:32.578 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607568' 2026-02-22T21:56:32.578 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607568 2026-02-22T21:56:32.579 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:56:32.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607568 -lt 51539607568 2026-02-22T21:56:32.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:32.782 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411342 2026-02-22T21:56:32.782 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:32.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:56:32.783 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411342 2026-02-22T21:56:32.783 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:32.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411342 2026-02-22T21:56:32.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411342' 2026-02-22T21:56:32.785 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 77309411342 2026-02-22T21:56:32.785 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:56:32.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411342 -lt 77309411342 2026-02-22T21:56:32.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:32.991 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-261993005059 2026-02-22T21:56:32.991 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:32.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:56:32.993 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-261993005059 2026-02-22T21:56:32.993 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:32.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=261993005059 2026-02-22T21:56:32.994 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 261993005059 2026-02-22T21:56:32.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 261993005059' 2026-02-22T21:56:32.995 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:56:33.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 261993005059 -lt 261993005059 2026-02-22T21:56:33.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:33.197 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-146028888076 2026-02-22T21:56:33.197 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:33.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:56:33.199 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-146028888076 2026-02-22T21:56:33.199 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:33.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=146028888076 2026-02-22T21:56:33.200 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 146028888076 2026-02-22T21:56:33.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 146028888076' 2026-02-22T21:56:33.201 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:56:33.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 146028888076 -lt 146028888076 2026-02-22T21:56:33.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:33.397 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-180388626441 2026-02-22T21:56:33.397 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:33.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:56:33.399 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-180388626441 2026-02-22T21:56:33.399 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:33.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=180388626441 2026-02-22T21:56:33.400 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 180388626441 2026-02-22T21:56:33.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 180388626441' 2026-02-22T21:56:33.401 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:56:33.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 180388626441 -lt 180388626441 2026-02-22T21:56:33.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:56:33.610 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-210453397511 2026-02-22T21:56:33.610 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:56:33.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:56:33.612 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-210453397511 2026-02-22T21:56:33.612 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:56:33.614 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=210453397511 2026-02-22T21:56:33.614 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 210453397511 2026-02-22T21:56:33.614 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 210453397511' 2026-02-22T21:56:33.614 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:56:33.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 210453397509 -lt 210453397511 2026-02-22T21:56:33.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:56:34.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:56:34.813 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:56:35.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 210453397512 -lt 210453397511 2026-02-22T21:56:35.018 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:56:35.018 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:56:35.018 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:56:35.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T21:56:35.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:56:35.283 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:56:35.283 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:56:35.283 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:56:35.283 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:56:35.283 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:56:35.283 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:56:35.476 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T21:56:35.476 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:56:35.476 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:56:35.476 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:56:35.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 5 2026-02-22T21:56:35.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 4 '!=' -1 2026-02-22T21:56:35.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: loop=0 2026-02-22T21:56:35.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: num_active_clean=4 2026-02-22T21:56:35.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T21:56:35.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.1 2026-02-22T21:56:35.839 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T21:56:35.839 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:56:35.839 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:56:35.839 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:56:35.839 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:56:35.839 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:56:35.840 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:56:35.840 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:56:36.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T21:56:36.036 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:56:36.037 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:56:36.037 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:56:36.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 5 2026-02-22T21:56:36.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 4 '!=' 4 2026-02-22T21:56:36.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T21:56:36.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T21:56:36.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T21:56:36.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T21:56:36.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T21:56:36.302 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T21:56:36.302 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T21:56:36.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T21:56:36.558 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T21:56:36.558 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: (( 1 >= 13 )) 2026-02-22T21:56:36.558 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T21:56:36.558 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.2 2026-02-22T21:56:36.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T21:56:36.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:56:36.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:56:36.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:56:36.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:56:36.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:56:36.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:56:36.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:56:36.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T21:56:36.952 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:56:36.952 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:56:36.953 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:56:37.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T21:56:37.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:56:37.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:56:37.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:383: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2026-02-22T21:56:37.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T21:56:37.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T21:56:37.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2026-02-22T21:56:37.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T21:56:37.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T21:56:37.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2026-02-22T21:56:37.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T21:56:37.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T21:56:37.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2026-02-22T21:56:37.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T21:56:37.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T21:56:37.474 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T21:56:37.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T21:56:37.784 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T21:56:37.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T21:56:37.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T21:56:37.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:56:37.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T21:56:37.795 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:56:37.795 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:56:37.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:56:37.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:56:37.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:56:37.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:56:37.907 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:56:37.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:56:37.908 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:56:37.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:56:37.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:56:37.909 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:56:37.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:56:37.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:56:37.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:56:37.911 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:56:37.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:56:37.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:56:37.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T21:56:37.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:56:37.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:37.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:37.936 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T21:56:37.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:56:37.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:56:37.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T21:56:37.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T21:56:37.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T21:56:37.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T21:56:37.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T21:56:37.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:56:37.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T21:56:37.940 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:56:37.940 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:56:37.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:56:37.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:56:37.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:56:37.941 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:56:37.941 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:56:37.941 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:56:37.942 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:56:37.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:56:37.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:56:37.944 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:56:37.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:56:37.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:56:37.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:56:37.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:56:37.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:56:37.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:56:37.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T21:56:37.948 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:56:37.949 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:37.949 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:37.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T21:56:37.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:56:37.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:56:37.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T21:56:37.951 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:56:37.951 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:37.951 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:37.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T21:56:37.954 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:56:37.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:56:37.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:56:37.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T21:56:37.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T21:56:37.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T21:56:37.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:56:37.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:56:37.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:56:37.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T21:56:37.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T21:56:37.976 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:56:37.977 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:56:37.977 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:56:37.977 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:56:37.977 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:37.977 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:37.977 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:37.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T21:56:38.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:56:38.009 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:56:38.009 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:56:38.009 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:56:38.010 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:56:38.010 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:56:38.010 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:56:38.010 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:56:38.010 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:56:38.010 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:56:38.010 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:38.011 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:38.011 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:56:38.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:56:38.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T21:56:38.063 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:56:38.063 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:56:38.063 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:56:38.063 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:56:38.064 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:56:38.064 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:56:38.064 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:56:38.064 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:56:38.064 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:56:38.065 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:38.065 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:38.065 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:56:38.065 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:56:38.065 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T21:56:38.119 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T21:56:38.119 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T21:56:38.119 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:56:38.119 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:56:38.119 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:56:38.119 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T21:56:38.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:56:38.231 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:56:38.231 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:56:38.231 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:56:38.231 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:56:38.231 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:38.231 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:38.231 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:38.232 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:56:38.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:56:38.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T21:56:38.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T21:56:38.357 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T21:56:38.366 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:56:39.369 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T21:56:39.369 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:56:39.369 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:56:39.369 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:56:39.370 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:39.370 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:39.370 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:56:39.370 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T21:56:39.370 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T21:56:39.417 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:56:39.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T21:56:39.424 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:56:37.993+0000 7f3800f8fec0 0 load: jerasure load: lrc 2026-02-22T21:56:39.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2026-02-22T21:56:39.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2026-02-22T21:56:39.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_recovery_multiple_errors: local objname=myobject 2026-02-22T21:56:39.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_recovery_multiple_errors: setup_osds 9 2026-02-22T21:56:39.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2026-02-22T21:56:39.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T21:56:39.426 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2026-02-22T21:56:39.427 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2026-02-22T21:56:39.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:56:39.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T21:56:39.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:56:39.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:56:39.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:56:39.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:56:39.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T21:56:39.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:56:39.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:56:39.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:56:39.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:56:39.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T21:56:39.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T21:56:39.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:56:39.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:56:39.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:56:39.452 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:56:39.452 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:56:39.452 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:56:39.453 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:56:39.453 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:39.453 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:39.453 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:39.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:39.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:56:39.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:56:39.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:56:39.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:56:39.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:56:39.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:56:39.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:56:39.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:56:39.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:56:39.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:56:39.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:56:39.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T21:56:39.456 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:56:39.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=78b7e980-6ff5-476c-8e1f-99cc522b18df 2026-02-22T21:56:39.456 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 78b7e980-6ff5-476c-8e1f-99cc522b18df 2026-02-22T21:56:39.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 78b7e980-6ff5-476c-8e1f-99cc522b18df' 2026-02-22T21:56:39.458 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:56:39.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXe5tpPu96GhAAx/8EDyN2TSG7TGyBbVSMzA== 2026-02-22T21:56:39.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXe5tpPu96GhAAx/8EDyN2TSG7TGyBbVSMzA=="}' 2026-02-22T21:56:39.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 78b7e980-6ff5-476c-8e1f-99cc522b18df -i td/test-erasure-eio/0/new.json 2026-02-22T21:56:39.551 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:56:39.558 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T21:56:39.559 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXe5tpPu96GhAAx/8EDyN2TSG7TGyBbVSMzA== --osd-uuid 78b7e980-6ff5-476c-8e1f-99cc522b18df 2026-02-22T21:56:39.574 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:39.573+0000 7f7b41db68c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:39.576 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:39.573+0000 7f7b41db68c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:39.577 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:39.573+0000 7f7b41db68c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:39.577 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:39.573+0000 7f7b41db68c0 -1 bdev(0x56222c7c1c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:56:39.577 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:39.573+0000 7f7b41db68c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T21:56:42.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T21:56:42.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:56:42.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:56:42.136 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:56:42.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:56:42.403 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:56:42.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:56:42.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:56:42.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:56:42.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:56:42.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:56:42.419 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:42.417+0000 7f97c011d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:42.420 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:42.417+0000 7f97c011d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:42.421 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:42.417+0000 7f97c011d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:42.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:56:42.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:56:42.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:56:42.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:56:42.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:56:42.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:42.597 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:56:42.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:56:42.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:42.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:56:42.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:43.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:43.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:43.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:56:43.800 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:56:43.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:43.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:56:43.943 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:43.941+0000 7f97c011d8c0 -1 Falling back to public interface 2026-02-22T21:56:44.001 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:45.003 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:45.003 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:45.003 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:56:45.003 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:56:45.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:45.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:56:45.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:46.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:46.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:46.204 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:56:46.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:56:46.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:46.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:56:46.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:47.015 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:47.001+0000 7f97c011d8c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:56:47.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:47.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:47.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:56:47.407 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:56:47.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:47.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:56:47.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:48.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:48.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:48.603 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:56:48.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:56:48.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:48.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:56:48.794 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/601758941,v1:127.0.0.1:6803/601758941] [v2:127.0.0.1:6804/601758941,v1:127.0.0.1:6805/601758941] exists,up 78b7e980-6ff5-476c-8e1f-99cc522b18df 2026-02-22T21:56:48.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:56:48.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:56:48.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:56:48.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:56:48.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T21:56:48.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:56:48.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:56:48.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T21:56:48.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:56:48.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:56:48.799 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:56:48.800 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:48.800 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:48.800 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:48.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:48.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:56:48.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:56:48.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:56:48.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:56:48.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:56:48.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:56:48.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:56:48.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:56:48.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:56:48.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:56:48.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:56:48.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T21:56:48.801 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:56:48.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d4411663-b523-4296-9385-ec7255476d7a 2026-02-22T21:56:48.802 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 d4411663-b523-4296-9385-ec7255476d7a 2026-02-22T21:56:48.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d4411663-b523-4296-9385-ec7255476d7a' 2026-02-22T21:56:48.802 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:56:48.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCge5tp6CBkMBAAWPvIV7bQQm+VcTQb/SPchg== 2026-02-22T21:56:48.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCge5tp6CBkMBAAWPvIV7bQQm+VcTQb/SPchg=="}' 2026-02-22T21:56:48.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d4411663-b523-4296-9385-ec7255476d7a -i td/test-erasure-eio/1/new.json 2026-02-22T21:56:49.004 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:56:49.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T21:56:49.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCge5tp6CBkMBAAWPvIV7bQQm+VcTQb/SPchg== --osd-uuid d4411663-b523-4296-9385-ec7255476d7a 2026-02-22T21:56:49.038 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:49.037+0000 7fc09f11d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:49.039 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:49.037+0000 7fc09f11d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:49.041 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:49.037+0000 7fc09f11d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:49.041 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:49.037+0000 7fc09f11d8c0 -1 bdev(0x557e5f1d1c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:56:49.042 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:49.041+0000 7fc09f11d8c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T21:56:51.563 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T21:56:51.563 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:56:51.564 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T21:56:51.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:56:51.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:56:51.826 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T21:56:51.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T21:56:51.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:56:51.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:56:51.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:56:51.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:56:51.842 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:51.841+0000 7ff82a24e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:51.843 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:51.841+0000 7ff82a24e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:51.845 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:51.841+0000 7ff82a24e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:52.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T21:56:52.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:56:52.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T21:56:52.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:56:52.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:56:52.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:52.037 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:56:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:56:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:52.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:56:52.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:53.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:53.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:53.235 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:56:53.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:56:53.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:53.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:56:53.355 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:53.353+0000 7ff82a24e8c0 -1 Falling back to public interface 2026-02-22T21:56:53.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:54.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:54.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:54.430 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:56:54.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:56:54.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:54.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:56:54.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:55.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:55.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:55.624 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:56:55.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:56:55.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:55.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:56:55.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:56:56.362 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:56.361+0000 7ff82a24e8c0 -1 osd.1 0 log_to_monitors true 2026-02-22T21:56:56.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:56:56.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:56:56.820 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:56:56.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:56:56.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:56:56.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:56:57.014 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3670608827,v1:127.0.0.1:6811/3670608827] [v2:127.0.0.1:6812/3670608827,v1:127.0.0.1:6813/3670608827] exists,up d4411663-b523-4296-9385-ec7255476d7a 2026-02-22T21:56:57.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:56:57.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:56:57.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:56:57.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:56:57.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T21:56:57.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:56:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:56:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T21:56:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:56:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T21:56:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:56:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:56:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:56:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:56:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T21:56:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T21:56:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:56:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:56:57.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:56:57.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:56:57.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:56:57.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:56:57.019 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:56:57.019 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:56:57.019 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:56:57.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:57.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:56:57.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:56:57.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:56:57.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:56:57.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:56:57.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:56:57.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:56:57.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:56:57.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:56:57.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:56:57.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:56:57.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:56:57.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T21:56:57.021 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:56:57.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e7f8a021-c9b3-4220-86ad-88fab4250de3 2026-02-22T21:56:57.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e7f8a021-c9b3-4220-86ad-88fab4250de3' 2026-02-22T21:56:57.022 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 e7f8a021-c9b3-4220-86ad-88fab4250de3 2026-02-22T21:56:57.022 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:56:57.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpe5tpZRf8ARAA11bYQq+2j752ABJ7b5tOrg== 2026-02-22T21:56:57.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpe5tpZRf8ARAA11bYQq+2j752ABJ7b5tOrg=="}' 2026-02-22T21:56:57.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e7f8a021-c9b3-4220-86ad-88fab4250de3 -i td/test-erasure-eio/2/new.json 2026-02-22T21:56:57.231 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:56:57.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T21:56:57.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCpe5tpZRf8ARAA11bYQq+2j752ABJ7b5tOrg== --osd-uuid e7f8a021-c9b3-4220-86ad-88fab4250de3 2026-02-22T21:56:57.257 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:57.253+0000 7f4521f6f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:57.259 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:57.257+0000 7f4521f6f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:57.260 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:57.257+0000 7f4521f6f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:57.260 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:57.257+0000 7f4521f6f8c0 -1 bdev(0x555d35095c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:56:57.260 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:57.257+0000 7f4521f6f8c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T21:56:59.542 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T21:56:59.542 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:56:59.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:56:59.543 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T21:56:59.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:56:59.820 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T21:56:59.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T21:56:59.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:56:59.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:56:59.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:56:59.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:56:59.835 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:59.833+0000 7fbe203e18c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:59.836 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:59.833+0000 7fbe203e18c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:56:59.837 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:56:59.833+0000 7fbe203e18c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:00.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T21:57:00.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:57:00.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T21:57:00.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:57:00.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:57:00.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:00.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:57:00.021 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:57:00.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:00.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:57:00.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:01.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:01.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:01.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:57:01.217 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:57:01.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:01.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:57:01.331 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:01.329+0000 7fbe203e18c0 -1 Falling back to public interface 2026-02-22T21:57:01.411 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:02.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:02.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:02.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:57:02.413 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:57:02.414 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:02.414 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:57:02.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:03.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:03.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:03.606 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:57:03.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:57:03.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:03.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:57:03.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:04.343 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:04.341+0000 7fbe203e18c0 -1 osd.2 0 log_to_monitors true 2026-02-22T21:57:04.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:04.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:04.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:57:04.808 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:57:04.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:04.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:57:05.009 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1695462700,v1:127.0.0.1:6819/1695462700] [v2:127.0.0.1:6820/1695462700,v1:127.0.0.1:6821/1695462700] exists,up e7f8a021-c9b3-4220-86ad-88fab4250de3 2026-02-22T21:57:05.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:57:05.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:57:05.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:57:05.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:57:05.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T21:57:05.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:57:05.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:57:05.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T21:57:05.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:57:05.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T21:57:05.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:57:05.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:57:05.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:57:05.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:57:05.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T21:57:05.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T21:57:05.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:57:05.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:57:05.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:57:05.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:57:05.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:57:05.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:57:05.015 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:57:05.016 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:57:05.016 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:57:05.016 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:57:05.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:57:05.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:57:05.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:57:05.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:57:05.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:57:05.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:57:05.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:57:05.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:57:05.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:57:05.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:57:05.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:57:05.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:57:05.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T21:57:05.018 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:57:05.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=332268d4-2f32-4da0-9abc-6eea2d8445a4 2026-02-22T21:57:05.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 332268d4-2f32-4da0-9abc-6eea2d8445a4' 2026-02-22T21:57:05.018 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 332268d4-2f32-4da0-9abc-6eea2d8445a4 2026-02-22T21:57:05.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:57:05.027 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxe5tpaqy3ARAAtaEVgT8oxjmZ0TywVRB1vA== 2026-02-22T21:57:05.027 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxe5tpaqy3ARAAtaEVgT8oxjmZ0TywVRB1vA=="}' 2026-02-22T21:57:05.027 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 332268d4-2f32-4da0-9abc-6eea2d8445a4 -i td/test-erasure-eio/3/new.json 2026-02-22T21:57:05.232 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:57:05.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T21:57:05.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxe5tpaqy3ARAAtaEVgT8oxjmZ0TywVRB1vA== --osd-uuid 332268d4-2f32-4da0-9abc-6eea2d8445a4 2026-02-22T21:57:05.258 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:05.257+0000 7fe885b7c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:05.261 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:05.257+0000 7fe885b7c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:05.262 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:05.261+0000 7fe885b7c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:05.262 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:05.261+0000 7fe885b7c8c0 -1 bdev(0x55f5d8c7bc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T21:57:05.263 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:05.261+0000 7fe885b7c8c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T21:57:07.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T21:57:07.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:57:07.537 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T21:57:07.538 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T21:57:07.538 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:57:07.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T21:57:07.802 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T21:57:07.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:57:07.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:57:07.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:57:07.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:57:07.825 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:07.821+0000 7f27107688c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:07.826 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:07.825+0000 7f27107688c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:07.827 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:07.825+0000 7f27107688c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:07.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T21:57:07.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:57:07.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T21:57:07.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:57:07.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:57:07.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:07.999 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:57:07.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:57:08.000 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:08.000 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:57:08.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:09.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:09.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:09.197 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:57:09.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:57:09.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:09.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:57:09.359 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:09.357+0000 7f27107688c0 -1 Falling back to public interface 2026-02-22T21:57:09.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:10.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:10.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:10.392 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:57:10.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:57:10.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:10.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:57:10.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:11.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:11.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:11.590 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:57:11.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:57:11.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:11.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:57:11.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:12.581 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:12.577+0000 7f27107688c0 -1 osd.3 0 log_to_monitors true 2026-02-22T21:57:12.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:12.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:12.784 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:57:12.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:57:12.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:12.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:57:12.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:13.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:13.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:13.986 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:57:13.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:57:13.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:13.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:57:14.185 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1069676492,v1:127.0.0.1:6827/1069676492] [v2:127.0.0.1:6828/1069676492,v1:127.0.0.1:6829/1069676492] exists,up 332268d4-2f32-4da0-9abc-6eea2d8445a4 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:57:14.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:57:14.187 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:57:14.187 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:57:14.187 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:57:14.187 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:57:14.187 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:57:14.187 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:57:14.187 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:57:14.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:57:14.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:57:14.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:57:14.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:57:14.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:57:14.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:57:14.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:57:14.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:57:14.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:57:14.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:57:14.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:57:14.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:57:14.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-22T21:57:14.188 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:57:14.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=18e7b7a6-4798-49d6-beed-ac59e5bb631b 2026-02-22T21:57:14.189 INFO:tasks.workunit.client.0.trial034.stdout:add osd4 18e7b7a6-4798-49d6-beed-ac59e5bb631b 2026-02-22T21:57:14.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 18e7b7a6-4798-49d6-beed-ac59e5bb631b' 2026-02-22T21:57:14.190 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:57:14.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6e5tpmEA0DBAAQ/CjBjQGRW7YpAgOT6Hsww== 2026-02-22T21:57:14.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6e5tpmEA0DBAAQ/CjBjQGRW7YpAgOT6Hsww=="}' 2026-02-22T21:57:14.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 18e7b7a6-4798-49d6-beed-ac59e5bb631b -i td/test-erasure-eio/4/new.json 2026-02-22T21:57:14.404 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:57:14.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-22T21:57:14.414 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6e5tpmEA0DBAAQ/CjBjQGRW7YpAgOT6Hsww== --osd-uuid 18e7b7a6-4798-49d6-beed-ac59e5bb631b 2026-02-22T21:57:14.430 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:14.429+0000 7f362036b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:14.431 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:14.429+0000 7f362036b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:14.432 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:14.429+0000 7f362036b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:14.433 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:14.429+0000 7f362036b8c0 -1 bdev(0x55c088511c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-22T21:57:14.433 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:14.429+0000 7f362036b8c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-22T21:57:16.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-22T21:57:16.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:57:16.723 INFO:tasks.workunit.client.0.trial034.stdout:adding osd4 key to auth repository 2026-02-22T21:57:16.724 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2026-02-22T21:57:16.724 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:57:16.992 INFO:tasks.workunit.client.0.trial034.stdout:start osd.4 2026-02-22T21:57:16.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2026-02-22T21:57:16.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:57:16.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:57:16.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:57:16.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:57:17.014 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:17.013+0000 7fa9283538c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:17.016 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:17.013+0000 7fa9283538c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:17.017 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:17.013+0000 7fa9283538c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:17.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2026-02-22T21:57:17.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:57:17.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2026-02-22T21:57:17.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:57:17.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:57:17.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:17.195 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:57:17.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:57:17.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:17.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:57:17.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:18.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:18.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:18.392 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:57:18.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:57:18.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:18.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:57:18.507 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:18.505+0000 7fa9283538c0 -1 Falling back to public interface 2026-02-22T21:57:18.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:19.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:19.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:19.586 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:57:19.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:57:19.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:19.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:57:19.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:20.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:20.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:20.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:57:20.780 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:57:20.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:20.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:57:20.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:21.560 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:21.557+0000 7fa9283538c0 -1 osd.4 0 log_to_monitors true 2026-02-22T21:57:21.975 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:21.976 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:57:21.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:21.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:57:21.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:21.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:57:22.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:23.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:23.178 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:57:23.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:23.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:57:23.179 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:23.179 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:57:23.369 INFO:tasks.workunit.client.0.trial034.stdout:osd.4 up in weight 1 up_from 33 up_thru 34 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/233503747,v1:127.0.0.1:6835/233503747] [v2:127.0.0.1:6836/233503747,v1:127.0.0.1:6837/233503747] exists,up 18e7b7a6-4798-49d6-beed-ac59e5bb631b 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:57:23.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:57:23.372 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-22T21:57:23.372 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-22T21:57:23.372 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:57:23.372 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:57:23.372 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:57:23.372 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:57:23.372 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:57:23.372 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:57:23.372 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:57:23.372 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:57:23.372 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:57:23.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-22T21:57:23.374 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:57:23.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=71c8ed22-08f5-47b6-b954-6ce43e0cb638 2026-02-22T21:57:23.375 INFO:tasks.workunit.client.0.trial034.stdout:add osd5 71c8ed22-08f5-47b6-b954-6ce43e0cb638 2026-02-22T21:57:23.375 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 71c8ed22-08f5-47b6-b954-6ce43e0cb638' 2026-02-22T21:57:23.375 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:57:23.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDe5tpqtYvFxAAf7mecYx4tlyR+kPLAivCAQ== 2026-02-22T21:57:23.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDe5tpqtYvFxAAf7mecYx4tlyR+kPLAivCAQ=="}' 2026-02-22T21:57:23.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 71c8ed22-08f5-47b6-b954-6ce43e0cb638 -i td/test-erasure-eio/5/new.json 2026-02-22T21:57:23.586 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:57:23.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-22T21:57:23.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDe5tpqtYvFxAAf7mecYx4tlyR+kPLAivCAQ== --osd-uuid 71c8ed22-08f5-47b6-b954-6ce43e0cb638 2026-02-22T21:57:23.611 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:23.609+0000 7fc9271518c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:23.612 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:23.609+0000 7fc9271518c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:23.613 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:23.613+0000 7fc9271518c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:23.614 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:23.613+0000 7fc9271518c0 -1 bdev(0x5604fcb69c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-22T21:57:23.614 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:23.613+0000 7fc9271518c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-22T21:57:25.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-22T21:57:25.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:57:25.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2026-02-22T21:57:25.900 INFO:tasks.workunit.client.0.trial034.stdout:adding osd5 key to auth repository 2026-02-22T21:57:25.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:57:26.165 INFO:tasks.workunit.client.0.trial034.stdout:start osd.5 2026-02-22T21:57:26.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2026-02-22T21:57:26.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:57:26.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:57:26.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:57:26.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:57:26.188 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:26.185+0000 7ffb7d5668c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:26.190 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:26.189+0000 7ffb7d5668c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:26.191 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:26.189+0000 7ffb7d5668c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:26.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2026-02-22T21:57:26.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:57:26.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2026-02-22T21:57:26.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:57:26.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:57:26.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:26.369 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:57:26.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:57:26.370 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:26.370 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:57:26.562 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:27.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:27.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:27.564 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:57:27.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:57:27.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:27.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:57:27.719 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:27.717+0000 7ffb7d5668c0 -1 Falling back to public interface 2026-02-22T21:57:27.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:28.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:28.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:28.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:57:28.759 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:57:28.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:28.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:57:28.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:29.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:29.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:29.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:57:29.950 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:57:29.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:29.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:57:30.144 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:30.748 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:30.744+0000 7ffb7d5668c0 -1 osd.5 0 log_to_monitors true 2026-02-22T21:57:31.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:31.145 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:57:31.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:31.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:57:31.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:31.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:57:31.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:32.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:32.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:32.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:57:32.341 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:57:32.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:32.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:57:32.535 INFO:tasks.workunit.client.0.trial034.stdout:osd.5 up in weight 1 up_from 41 up_thru 42 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2911655032,v1:127.0.0.1:6843/2911655032] [v2:127.0.0.1:6844/2911655032,v1:127.0.0.1:6845/2911655032] exists,up 71c8ed22-08f5-47b6-b954-6ce43e0cb638 2026-02-22T21:57:32.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:57:32.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:57:32.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-22T21:57:32.538 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:57:32.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=00874c63-4d7f-4937-8475-dae3c796c998 2026-02-22T21:57:32.539 INFO:tasks.workunit.client.0.trial034.stdout:add osd6 00874c63-4d7f-4937-8475-dae3c796c998 2026-02-22T21:57:32.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 00874c63-4d7f-4937-8475-dae3c796c998' 2026-02-22T21:57:32.540 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:57:32.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMe5tpsikKIRAAXhzLj2Gje7Sua/rdftZi+g== 2026-02-22T21:57:32.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMe5tpsikKIRAAXhzLj2Gje7Sua/rdftZi+g=="}' 2026-02-22T21:57:32.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 00874c63-4d7f-4937-8475-dae3c796c998 -i td/test-erasure-eio/6/new.json 2026-02-22T21:57:32.753 INFO:tasks.workunit.client.0.trial034.stdout:6 2026-02-22T21:57:32.763 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-22T21:57:32.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMe5tpsikKIRAAXhzLj2Gje7Sua/rdftZi+g== --osd-uuid 00874c63-4d7f-4937-8475-dae3c796c998 2026-02-22T21:57:32.779 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:32.776+0000 7f8eda1228c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:32.780 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:32.776+0000 7f8eda1228c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:32.781 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:32.780+0000 7f8eda1228c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:32.781 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:32.780+0000 7f8eda1228c0 -1 bdev(0x56473f9f9c00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-22T21:57:32.782 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:32.780+0000 7f8eda1228c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-22T21:57:35.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-22T21:57:35.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:57:35.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2026-02-22T21:57:35.047 INFO:tasks.workunit.client.0.trial034.stdout:adding osd6 key to auth repository 2026-02-22T21:57:35.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:57:35.308 INFO:tasks.workunit.client.0.trial034.stdout:start osd.6 2026-02-22T21:57:35.309 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2026-02-22T21:57:35.309 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:57:35.309 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:57:35.310 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:57:35.310 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:57:35.331 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:35.328+0000 7fdfea6a88c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:35.332 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:35.328+0000 7fdfea6a88c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:35.333 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:35.332+0000 7fdfea6a88c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:35.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2026-02-22T21:57:35.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:57:35.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2026-02-22T21:57:35.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:57:35.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:57:35.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:35.507 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:57:35.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:57:35.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:35.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:57:35.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:36.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:36.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:36.702 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:57:36.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:57:36.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:36.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:57:36.883 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:36.880+0000 7fdfea6a88c0 -1 Falling back to public interface 2026-02-22T21:57:36.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:37.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:37.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:37.898 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:57:37.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:57:37.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:37.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:57:38.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:39.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:39.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:39.094 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:57:39.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:57:39.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:39.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:57:39.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:39.854 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:39.852+0000 7fdfea6a88c0 -1 osd.6 0 log_to_monitors true 2026-02-22T21:57:40.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:40.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:40.289 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:57:40.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:57:40.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:40.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:57:40.487 INFO:tasks.workunit.client.0.trial034.stdout:osd.6 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2794997930,v1:127.0.0.1:6851/2794997930] [v2:127.0.0.1:6852/2794997930,v1:127.0.0.1:6853/2794997930] exists,up 00874c63-4d7f-4937-8475-dae3c796c998 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/7 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:57:40.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:57:40.489 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:57:40.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/7 2026-02-22T21:57:40.491 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:57:40.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=db297b7b-db0e-4e0b-ad3c-9c0fda9dcafb 2026-02-22T21:57:40.491 INFO:tasks.workunit.client.0.trial034.stdout:add osd7 db297b7b-db0e-4e0b-ad3c-9c0fda9dcafb 2026-02-22T21:57:40.492 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 db297b7b-db0e-4e0b-ad3c-9c0fda9dcafb' 2026-02-22T21:57:40.492 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:57:40.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUe5tpd+otHhAAJKq6wcduIFgzy9PcbOxCrA== 2026-02-22T21:57:40.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUe5tpd+otHhAAJKq6wcduIFgzy9PcbOxCrA=="}' 2026-02-22T21:57:40.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new db297b7b-db0e-4e0b-ad3c-9c0fda9dcafb -i td/test-erasure-eio/7/new.json 2026-02-22T21:57:40.709 INFO:tasks.workunit.client.0.trial034.stdout:7 2026-02-22T21:57:40.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/7/new.json 2026-02-22T21:57:40.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUe5tpd+otHhAAJKq6wcduIFgzy9PcbOxCrA== --osd-uuid db297b7b-db0e-4e0b-ad3c-9c0fda9dcafb 2026-02-22T21:57:40.735 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:40.732+0000 7f54c91308c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:40.737 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:40.732+0000 7f54c91308c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:40.738 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:40.736+0000 7f54c91308c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:40.738 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:40.736+0000 7f54c91308c0 -1 bdev(0x563257055c00 td/test-erasure-eio/7/block) open stat got: (1) Operation not permitted 2026-02-22T21:57:40.738 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:40.736+0000 7f54c91308c0 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2026-02-22T21:57:43.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2026-02-22T21:57:43.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:57:43.039 INFO:tasks.workunit.client.0.trial034.stdout:adding osd7 key to auth repository 2026-02-22T21:57:43.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2026-02-22T21:57:43.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:57:43.304 INFO:tasks.workunit.client.0.trial034.stdout:start osd.7 2026-02-22T21:57:43.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2026-02-22T21:57:43.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:57:43.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:57:43.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:57:43.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:57:43.327 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:43.324+0000 7ff3fcb5e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:43.329 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:43.324+0000 7ff3fcb5e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:43.330 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:43.328+0000 7ff3fcb5e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:43.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2026-02-22T21:57:43.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:57:43.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2026-02-22T21:57:43.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:57:43.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:57:43.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:43.502 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:57:43.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:57:43.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:43.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2026-02-22T21:57:43.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:44.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:44.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:44.695 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:57:44.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:57:44.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:44.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2026-02-22T21:57:44.831 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:44.828+0000 7ff3fcb5e8c0 -1 Falling back to public interface 2026-02-22T21:57:44.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:45.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:45.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:45.889 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:57:45.890 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:57:45.890 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:45.890 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2026-02-22T21:57:46.084 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:47.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:47.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:47.085 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:57:47.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:57:47.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:47.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2026-02-22T21:57:47.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:48.054 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:48.052+0000 7ff3fcb5e8c0 -1 osd.7 0 log_to_monitors true 2026-02-22T21:57:48.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:48.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:48.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:57:48.280 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:57:48.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:48.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2026-02-22T21:57:48.476 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:49.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:49.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:49.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:57:49.477 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:57:49.478 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:49.478 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2026-02-22T21:57:49.669 INFO:tasks.workunit.client.0.trial034.stdout:osd.7 up in weight 1 up_from 58 up_thru 60 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1549016653,v1:127.0.0.1:6859/1549016653] [v2:127.0.0.1:6860/1549016653,v1:127.0.0.1:6861/1549016653] exists,up db297b7b-db0e-4e0b-ad3c-9c0fda9dcafb 2026-02-22T21:57:49.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/8 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:57:49.670 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:57:49.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/8 2026-02-22T21:57:49.672 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:57:49.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=399c932f-ce36-4521-ac51-add7b3bc0934 2026-02-22T21:57:49.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 399c932f-ce36-4521-ac51-add7b3bc0934' 2026-02-22T21:57:49.673 INFO:tasks.workunit.client.0.trial034.stdout:add osd8 399c932f-ce36-4521-ac51-add7b3bc0934 2026-02-22T21:57:49.673 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:57:49.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDde5tpREHvKBAACT275T/VdeYqS2djDCGCFw== 2026-02-22T21:57:49.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDde5tpREHvKBAACT275T/VdeYqS2djDCGCFw=="}' 2026-02-22T21:57:49.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 399c932f-ce36-4521-ac51-add7b3bc0934 -i td/test-erasure-eio/8/new.json 2026-02-22T21:57:49.894 INFO:tasks.workunit.client.0.trial034.stdout:8 2026-02-22T21:57:49.903 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/8/new.json 2026-02-22T21:57:49.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDde5tpREHvKBAACT275T/VdeYqS2djDCGCFw== --osd-uuid 399c932f-ce36-4521-ac51-add7b3bc0934 2026-02-22T21:57:49.927 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:49.924+0000 7f49b21428c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:49.929 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:49.924+0000 7f49b21428c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:49.930 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:49.928+0000 7f49b21428c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:49.931 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:49.928+0000 7f49b21428c0 -1 bdev(0x564032101c00 td/test-erasure-eio/8/block) open stat got: (1) Operation not permitted 2026-02-22T21:57:49.931 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:49.928+0000 7f49b21428c0 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2026-02-22T21:57:52.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2026-02-22T21:57:52.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:57:52.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2026-02-22T21:57:52.230 INFO:tasks.workunit.client.0.trial034.stdout:adding osd8 key to auth repository 2026-02-22T21:57:52.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:57:52.494 INFO:tasks.workunit.client.0.trial034.stdout:start osd.8 2026-02-22T21:57:52.495 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2026-02-22T21:57:52.495 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:57:52.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:57:52.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:57:52.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:57:52.517 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:52.512+0000 7fd60563e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:52.519 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:52.516+0000 7fd60563e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:52.520 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:52.516+0000 7fd60563e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:57:52.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2026-02-22T21:57:52.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:57:52.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2026-02-22T21:57:52.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:57:52.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:57:52.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:52.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:57:52.701 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:57:52.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:52.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2026-02-22T21:57:52.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:53.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:53.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:53.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:57:53.902 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:57:53.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:53.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2026-02-22T21:57:54.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:54.259 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:54.260+0000 7fd60563e8c0 -1 Falling back to public interface 2026-02-22T21:57:55.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:55.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:55.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:57:55.106 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:57:55.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:55.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2026-02-22T21:57:55.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:56.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:56.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:56.298 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:57:56.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:57:56.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:56.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2026-02-22T21:57:56.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:57.252 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:57:57.252+0000 7fd60563e8c0 -1 osd.8 0 log_to_monitors true 2026-02-22T21:57:57.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:57.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:57.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:57:57.500 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:57:57.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:57.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2026-02-22T21:57:57.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:57:58.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:57:58.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:57:58.712 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:57:58.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:57:58.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:57:58.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2026-02-22T21:57:58.908 INFO:tasks.workunit.client.0.trial034.stdout:osd.8 up in weight 1 up_from 67 up_thru 69 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/2219528270,v1:127.0.0.1:6867/2219528270] [v2:127.0.0.1:6868/2219528270,v1:127.0.0.1:6869/2219528270] exists,up 399c932f-ce36-4521-ac51-add7b3bc0934 2026-02-22T21:57:58.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:57:58.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:57:58.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:57:58.910 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T21:57:58.910 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:57:58.910 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:57:58.910 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:57:58.910 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:57:58.910 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:57:58.911 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T21:57:58.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T21:57:58.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T21:57:58.972 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:57:58.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T21:57:58.980 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:56:43.945+0000 7f97c011d8c0 0 load: jerasure load: lrc 2026-02-22T21:57:58.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:423: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2026-02-22T21:57:58.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2026-02-22T21:57:58.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T21:57:58.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T21:57:58.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2026-02-22T21:57:58.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T21:57:58.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2026-02-22T21:57:58.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T21:57:58.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2026-02-22T21:57:59.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T21:57:59.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T21:57:59.585 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T21:57:59.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:58:00.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T21:58:00.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:58:00.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:58:00.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:58:00.596 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:58:00.597 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:58:00.597 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:58:00.598 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:58:00.598 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:58:00.598 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:58:00.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:58:00.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:58:00.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:58:00.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:58:00.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:58:00.707 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:58:00.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:58:00.911 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:58:00.911 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:58:00.912 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:58:00.912 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:58:00.912 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:58:00.912 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:58:00.912 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:58:00.912 INFO:tasks.workunit.client.0.trial034.stderr:8' 2026-02-22T21:58:00.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:58:00.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:00.912 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:58:00.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803793 2026-02-22T21:58:00.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803793 2026-02-22T21:58:00.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793' 2026-02-22T21:58:00.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:00.985 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:58:01.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607567 2026-02-22T21:58:01.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607567 2026-02-22T21:58:01.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567' 2026-02-22T21:58:01.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:01.061 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:58:01.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=73014444046 2026-02-22T21:58:01.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 73014444046 2026-02-22T21:58:01.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-73014444046' 2026-02-22T21:58:01.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:01.135 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:58:01.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182412 2026-02-22T21:58:01.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182412 2026-02-22T21:58:01.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-73014444046 3-107374182412' 2026-02-22T21:58:01.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:01.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:58:01.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=141733920778 2026-02-22T21:58:01.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 141733920778 2026-02-22T21:58:01.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-73014444046 3-107374182412 4-141733920778' 2026-02-22T21:58:01.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:01.285 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:58:01.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=176093659144 2026-02-22T21:58:01.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 176093659144 2026-02-22T21:58:01.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-73014444046 3-107374182412 4-141733920778 5-176093659144' 2026-02-22T21:58:01.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:01.360 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:58:01.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=210453397511 2026-02-22T21:58:01.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 210453397511 2026-02-22T21:58:01.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-73014444046 3-107374182412 4-141733920778 5-176093659144 6-210453397511' 2026-02-22T21:58:01.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:01.434 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T21:58:01.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=249108103173 2026-02-22T21:58:01.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 249108103173 2026-02-22T21:58:01.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-73014444046 3-107374182412 4-141733920778 5-176093659144 6-210453397511 7-249108103173' 2026-02-22T21:58:01.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:01.515 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T21:58:01.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=287762808835 2026-02-22T21:58:01.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 287762808835 2026-02-22T21:58:01.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-73014444046 3-107374182412 4-141733920778 5-176093659144 6-210453397511 7-249108103173 8-287762808835' 2026-02-22T21:58:01.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:01.592 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803793 2026-02-22T21:58:01.592 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:01.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:58:01.593 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803793 2026-02-22T21:58:01.593 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:01.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803793 2026-02-22T21:58:01.595 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803793 2026-02-22T21:58:01.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803793' 2026-02-22T21:58:01.595 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:58:01.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803791 -lt 25769803793 2026-02-22T21:58:01.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:58:02.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:58:02.786 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:58:03.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803793 -lt 25769803793 2026-02-22T21:58:03.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:03.006 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607567 2026-02-22T21:58:03.006 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:03.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:58:03.009 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607567 2026-02-22T21:58:03.009 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:03.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607567 2026-02-22T21:58:03.010 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607567 2026-02-22T21:58:03.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607567' 2026-02-22T21:58:03.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:58:03.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607567 -lt 51539607567 2026-02-22T21:58:03.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:03.208 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-73014444046 2026-02-22T21:58:03.208 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:03.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:58:03.209 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-73014444046 2026-02-22T21:58:03.209 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:03.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=73014444046 2026-02-22T21:58:03.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 73014444046' 2026-02-22T21:58:03.211 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 73014444046 2026-02-22T21:58:03.211 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:58:03.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 73014444046 -lt 73014444046 2026-02-22T21:58:03.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:03.407 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182412 2026-02-22T21:58:03.408 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:03.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:58:03.409 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182412 2026-02-22T21:58:03.409 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:03.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182412 2026-02-22T21:58:03.411 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 107374182412 2026-02-22T21:58:03.411 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182412' 2026-02-22T21:58:03.411 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:58:03.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182412 -lt 107374182412 2026-02-22T21:58:03.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:03.615 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-141733920778 2026-02-22T21:58:03.616 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:03.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:58:03.617 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-141733920778 2026-02-22T21:58:03.617 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:03.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=141733920778 2026-02-22T21:58:03.619 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 141733920778 2026-02-22T21:58:03.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 141733920778' 2026-02-22T21:58:03.619 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:58:03.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 141733920778 -lt 141733920778 2026-02-22T21:58:03.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:03.821 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-176093659144 2026-02-22T21:58:03.821 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:03.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:58:03.823 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-176093659144 2026-02-22T21:58:03.823 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:03.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=176093659144 2026-02-22T21:58:03.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 176093659144' 2026-02-22T21:58:03.825 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 176093659144 2026-02-22T21:58:03.825 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:58:04.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 176093659145 -lt 176093659144 2026-02-22T21:58:04.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:04.036 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-210453397511 2026-02-22T21:58:04.036 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:04.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:58:04.038 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-210453397511 2026-02-22T21:58:04.038 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:04.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=210453397511 2026-02-22T21:58:04.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 210453397511' 2026-02-22T21:58:04.040 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 210453397511 2026-02-22T21:58:04.040 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:58:04.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 210453397511 -lt 210453397511 2026-02-22T21:58:04.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:04.235 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 7-249108103173 2026-02-22T21:58:04.235 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:04.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=7 2026-02-22T21:58:04.236 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 7-249108103173 2026-02-22T21:58:04.236 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:04.238 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=249108103173 2026-02-22T21:58:04.238 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.7 seq 249108103173' 2026-02-22T21:58:04.238 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.7 seq 249108103173 2026-02-22T21:58:04.239 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T21:58:04.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 249108103173 -lt 249108103173 2026-02-22T21:58:04.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:04.442 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 8-287762808835 2026-02-22T21:58:04.442 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:04.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=8 2026-02-22T21:58:04.444 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 8-287762808835 2026-02-22T21:58:04.444 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:04.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=287762808835 2026-02-22T21:58:04.446 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.8 seq 287762808835 2026-02-22T21:58:04.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.8 seq 287762808835' 2026-02-22T21:58:04.446 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T21:58:04.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 287762808836 -lt 287762808835 2026-02-22T21:58:04.642 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:58:04.642 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:58:04.642 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:58:04.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T21:58:04.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:58:04.904 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:58:04.904 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:58:04.904 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:58:04.905 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:58:04.905 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:58:04.905 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:58:05.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T21:58:05.104 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:58:05.105 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:58:05.105 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:58:05.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T21:58:05.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:58:05.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:58:05.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-22T21:58:05.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T21:58:05.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T21:58:05.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-22T21:58:05.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:58:05.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T21:58:05.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:58:05.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T21:58:05.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:58:05.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T21:58:05.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:58:05.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T21:58:05.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-22T21:58:05.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:427: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2026-02-22T21:58:05.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T21:58:05.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T21:58:05.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T21:58:05.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T21:58:05.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T21:58:05.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T21:58:05.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=myobject 2026-02-22T21:58:05.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T21:58:05.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T21:58:05.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T21:58:05.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=0 2026-02-22T21:58:05.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T21:58:05.388 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure myobject 2026-02-22T21:58:05.388 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:58:05.388 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2026-02-22T21:58:05.388 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-22T21:58:05.389 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:58:05.586 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T21:58:05.586 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:58:05.586 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:58:05.586 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:58:05.586 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:58:05.587 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:58:05.587 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:58:05.587 INFO:tasks.workunit.client.0.trial034.stderr:7' 2026-02-22T21:58:05.587 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2026-02-22T21:58:05.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2026-02-22T21:58:05.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T21:58:05.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=3 2026-02-22T21:58:05.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T21:58:05.587 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/3/type 2026-02-22T21:58:05.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T21:58:05.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-22T21:58:05.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T21:58:05.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2026-02-22T21:58:05.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T21:58:05.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T21:58:05.590 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T21:58:05.590 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2026-02-22T21:58:05.590 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T21:58:05.590 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T21:58:05.590 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:58:05.590 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:05.590 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:05.590 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T21:58:05.590 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-22T21:58:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T21:58:05.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T21:58:05.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T21:58:05.652 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.3 2026-02-22T21:58:05.652 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T21:58:05.652 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T21:58:05.652 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:58:05.652 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:05.652 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:05.652 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T21:58:05.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T21:58:05.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2026-02-22T21:58:05.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:430: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2026-02-22T21:58:05.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T21:58:05.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T21:58:05.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T21:58:05.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T21:58:05.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T21:58:05.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T21:58:05.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=myobject 2026-02-22T21:58:05.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T21:58:05.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T21:58:05.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T21:58:05.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=3 2026-02-22T21:58:05.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T21:58:05.713 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure myobject 2026-02-22T21:58:05.713 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:58:05.713 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2026-02-22T21:58:05.713 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-22T21:58:05.713 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:58:05.907 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T21:58:05.907 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:58:05.907 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:58:05.907 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:58:05.907 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:58:05.907 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:58:05.907 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:58:05.907 INFO:tasks.workunit.client.0.trial034.stderr:7' 2026-02-22T21:58:05.908 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2026-02-22T21:58:05.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2026-02-22T21:58:05.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T21:58:05.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=0 2026-02-22T21:58:05.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T21:58:05.908 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/0/type 2026-02-22T21:58:05.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T21:58:05.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-22T21:58:05.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T21:58:05.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2026-02-22T21:58:05.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T21:58:05.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T21:58:05.909 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T21:58:05.909 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2026-02-22T21:58:05.909 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:58:05.910 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:58:05.910 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:58:05.910 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:05.910 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:05.910 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T21:58:05.910 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-22T21:58:05.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T21:58:05.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T21:58:05.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T21:58:05.972 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.0 2026-02-22T21:58:05.973 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:58:05.973 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:58:05.974 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:58:05.974 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:05.974 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:05.974 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T21:58:05.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T21:58:05.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2026-02-22T21:58:06.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2026-02-22T21:58:06.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T21:58:06.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T21:58:06.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T21:58:06.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T21:58:06.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T21:58:06.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T21:58:06.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=myobject 2026-02-22T21:58:06.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T21:58:06.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T21:58:06.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T21:58:06.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=4 2026-02-22T21:58:06.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T21:58:06.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure myobject 2026-02-22T21:58:06.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:58:06.034 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2026-02-22T21:58:06.035 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-22T21:58:06.035 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:58:06.229 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T21:58:06.230 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:58:06.230 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:58:06.230 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:58:06.230 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:58:06.230 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:58:06.230 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:58:06.230 INFO:tasks.workunit.client.0.trial034.stderr:7' 2026-02-22T21:58:06.230 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2026-02-22T21:58:06.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2026-02-22T21:58:06.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T21:58:06.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=6 2026-02-22T21:58:06.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T21:58:06.231 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/6/type 2026-02-22T21:58:06.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T21:58:06.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2026-02-22T21:58:06.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T21:58:06.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=6 2026-02-22T21:58:06.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T21:58:06.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T21:58:06.233 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T21:58:06.233 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.6 2026-02-22T21:58:06.233 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2026-02-22T21:58:06.233 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2026-02-22T21:58:06.233 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:58:06.233 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:06.234 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:06.234 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.6.asok 2026-02-22T21:58:06.234 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2026-02-22T21:58:06.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T21:58:06.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T21:58:06.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T21:58:06.293 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.6 2026-02-22T21:58:06.293 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2026-02-22T21:58:06.293 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2026-02-22T21:58:06.293 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:58:06.293 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:06.293 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:06.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.6.asok 2026-02-22T21:58:06.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T21:58:06.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2026-02-22T21:58:06.352 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2026-02-22T21:58:06.352 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T21:58:06.352 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2026-02-22T21:58:06.353 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-22T21:58:06.353 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T21:58:06.552 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T21:58:06.552 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:58:06.552 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:58:06.552 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T21:58:06.552 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:58:06.552 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:58:06.552 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:58:06.552 INFO:tasks.workunit.client.0.trial034.stderr:7' 2026-02-22T21:58:06.552 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2026-02-22T21:58:06.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2026-02-22T21:58:06.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: local -a initial_osds 2026-02-22T21:58:06.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:434: TEST_ec_recovery_multiple_errors: local last_osd=7 2026-02-22T21:58:06.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:436: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2026-02-22T21:58:06.553 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:58:06.553 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:58:06.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:58:06.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:58:06.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:58:07.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:58:07.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:437: TEST_ec_recovery_multiple_errors: ceph osd down 7 2026-02-22T21:58:08.051 INFO:tasks.workunit.client.0.trial034.stderr:osd.7 is already down. 2026-02-22T21:58:08.060 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: ceph osd out 7 2026-02-22T21:58:08.306 INFO:tasks.workunit.client.0.trial034.stderr:osd.7 is already out. 2026-02-22T21:58:08.317 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: wait_for_clean 2026-02-22T21:58:08.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:58:08.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:58:08.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:58:08.318 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:58:08.318 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:58:08.318 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:58:08.319 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:58:08.319 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:58:08.319 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:58:08.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:58:08.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:58:08.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:58:08.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:58:08.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:58:08.427 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:58:08.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:58:08.632 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:58:08.632 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:58:08.632 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:58:08.632 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:58:08.632 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:58:08.632 INFO:tasks.workunit.client.0.trial034.stderr:6 2026-02-22T21:58:08.632 INFO:tasks.workunit.client.0.trial034.stderr:7 2026-02-22T21:58:08.633 INFO:tasks.workunit.client.0.trial034.stderr:8' 2026-02-22T21:58:08.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:58:08.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:08.633 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:58:08.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803797 2026-02-22T21:58:08.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803797 2026-02-22T21:58:08.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803797' 2026-02-22T21:58:08.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:08.705 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:58:08.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607571 2026-02-22T21:58:08.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607571 2026-02-22T21:58:08.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803797 1-51539607571' 2026-02-22T21:58:08.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:08.785 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:58:08.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=73014444049 2026-02-22T21:58:08.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 73014444049 2026-02-22T21:58:08.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803797 1-51539607571 2-73014444049' 2026-02-22T21:58:08.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:08.862 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:58:08.936 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182416 2026-02-22T21:58:08.936 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182416 2026-02-22T21:58:08.936 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803797 1-51539607571 2-73014444049 3-107374182416' 2026-02-22T21:58:08.936 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:08.936 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:58:09.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=141733920782 2026-02-22T21:58:09.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 141733920782 2026-02-22T21:58:09.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803797 1-51539607571 2-73014444049 3-107374182416 4-141733920782' 2026-02-22T21:58:09.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:09.013 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:58:09.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=176093659148 2026-02-22T21:58:09.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 176093659148 2026-02-22T21:58:09.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803797 1-51539607571 2-73014444049 3-107374182416 4-141733920782 5-176093659148' 2026-02-22T21:58:09.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:09.088 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:58:09.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=210453397514 2026-02-22T21:58:09.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 210453397514 2026-02-22T21:58:09.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803797 1-51539607571 2-73014444049 3-107374182416 4-141733920782 5-176093659148 6-210453397514' 2026-02-22T21:58:09.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:09.164 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T21:58:09.219 INFO:tasks.workunit.client.0.trial034.stderr:Error ENXIO: problem getting command descriptions from osd.7 2026-02-22T21:58:09.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq= 2026-02-22T21:58:09.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z '' 2026-02-22T21:58:09.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: continue 2026-02-22T21:58:09.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:58:09.222 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T21:58:09.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=287762808839 2026-02-22T21:58:09.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 287762808839 2026-02-22T21:58:09.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803797 1-51539607571 2-73014444049 3-107374182416 4-141733920782 5-176093659148 6-210453397514 8-287762808839' 2026-02-22T21:58:09.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:09.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803797 2026-02-22T21:58:09.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:09.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:58:09.299 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803797 2026-02-22T21:58:09.299 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:09.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803797 2026-02-22T21:58:09.300 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803797 2026-02-22T21:58:09.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803797' 2026-02-22T21:58:09.301 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:58:09.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803795 -lt 25769803797 2026-02-22T21:58:09.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:58:10.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:58:10.512 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:58:10.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803797 -lt 25769803797 2026-02-22T21:58:10.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:10.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607571 2026-02-22T21:58:10.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:10.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:58:10.714 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607571 2026-02-22T21:58:10.714 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:10.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607571 2026-02-22T21:58:10.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607571' 2026-02-22T21:58:10.716 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607571 2026-02-22T21:58:10.716 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:58:10.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607571 -lt 51539607571 2026-02-22T21:58:10.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:10.925 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-73014444049 2026-02-22T21:58:10.925 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:10.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:58:10.926 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-73014444049 2026-02-22T21:58:10.927 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:10.928 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=73014444049 2026-02-22T21:58:10.928 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 73014444049' 2026-02-22T21:58:10.929 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 73014444049 2026-02-22T21:58:10.929 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:58:11.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 73014444050 -lt 73014444049 2026-02-22T21:58:11.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:11.132 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182416 2026-02-22T21:58:11.132 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:11.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:58:11.134 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182416 2026-02-22T21:58:11.134 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:11.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182416 2026-02-22T21:58:11.136 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 107374182416 2026-02-22T21:58:11.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182416' 2026-02-22T21:58:11.136 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:58:11.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182416 -lt 107374182416 2026-02-22T21:58:11.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:11.347 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-141733920782 2026-02-22T21:58:11.348 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:11.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:58:11.349 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-141733920782 2026-02-22T21:58:11.349 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:11.351 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=141733920782 2026-02-22T21:58:11.351 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 141733920782' 2026-02-22T21:58:11.351 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 141733920782 2026-02-22T21:58:11.352 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:58:11.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 141733920782 -lt 141733920782 2026-02-22T21:58:11.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:11.555 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-176093659148 2026-02-22T21:58:11.555 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:11.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:58:11.557 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-176093659148 2026-02-22T21:58:11.557 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:11.559 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=176093659148 2026-02-22T21:58:11.559 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 176093659148' 2026-02-22T21:58:11.559 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 176093659148 2026-02-22T21:58:11.560 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:58:11.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 176093659148 -lt 176093659148 2026-02-22T21:58:11.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:11.761 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-210453397514 2026-02-22T21:58:11.761 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:11.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:58:11.761 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-210453397514 2026-02-22T21:58:11.762 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:11.762 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=210453397514 2026-02-22T21:58:11.762 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 210453397514' 2026-02-22T21:58:11.763 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 210453397514 2026-02-22T21:58:11.763 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:58:11.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 210453397514 -lt 210453397514 2026-02-22T21:58:11.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:58:11.973 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 8-287762808839 2026-02-22T21:58:11.973 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:58:11.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=8 2026-02-22T21:58:11.975 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 8-287762808839 2026-02-22T21:58:11.975 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:58:11.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=287762808839 2026-02-22T21:58:11.976 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.8 seq 287762808839 2026-02-22T21:58:11.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.8 seq 287762808839' 2026-02-22T21:58:11.977 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T21:58:12.184 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 287762808839 -lt 287762808839 2026-02-22T21:58:12.184 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:58:12.185 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:58:12.185 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:58:12.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T21:58:12.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:58:12.434 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:58:12.434 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:58:12.434 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:58:12.434 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:58:12.434 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:58:12.434 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:58:12.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T21:58:12.626 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:58:12.626 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:58:12.626 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:58:12.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T21:58:12.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:58:12.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:58:12.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2026-02-22T21:58:12.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T21:58:12.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T21:58:12.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2026-02-22T21:58:12.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T21:58:12.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T21:58:12.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2026-02-22T21:58:12.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T21:58:12.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T21:58:12.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:445: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2026-02-22T21:58:12.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T21:58:12.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T21:58:13.193 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T21:58:13.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T21:58:13.499 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T21:58:13.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T21:58:13.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T21:58:13.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:58:13.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T21:58:13.510 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:58:13.510 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:58:13.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:58:13.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:58:13.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:58:13.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:58:13.824 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:58:13.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:58:13.825 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:58:13.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:58:13.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:58:13.827 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:58:13.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:58:13.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:58:13.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:58:13.829 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:58:13.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:58:13.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:58:13.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T21:58:13.858 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:58:13.859 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:13.859 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:13.859 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T21:58:13.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:58:13.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:58:13.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T21:58:13.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T21:58:13.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T21:58:13.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T21:58:13.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T21:58:13.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:58:13.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T21:58:13.862 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:58:13.862 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:58:13.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:58:13.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:58:13.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:58:13.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:58:13.864 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:58:13.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:58:13.865 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:58:13.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:58:13.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:58:13.866 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:58:13.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:58:13.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:58:13.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:58:13.868 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:58:13.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:58:13.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:58:13.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T21:58:13.872 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:58:13.872 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:13.872 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:13.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T21:58:13.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:58:13.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:58:13.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T21:58:13.875 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:58:13.875 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:13.876 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:13.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T21:58:13.877 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:58:13.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:58:13.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:58:13.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T21:58:13.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T21:58:13.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T21:58:13.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:58:13.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:58:13.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:58:13.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T21:58:13.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T21:58:13.901 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:58:13.901 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:58:13.901 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:58:13.901 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:58:13.901 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:13.901 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:13.901 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:58:13.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T21:58:13.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:58:13.933 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:58:13.933 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:58:13.933 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:58:13.933 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:58:13.934 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:58:13.934 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:58:13.934 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:58:13.935 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:58:13.935 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:58:13.935 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:13.935 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:13.935 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:58:13.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:58:13.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T21:58:13.990 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:58:13.990 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:58:13.990 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:58:13.991 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:58:13.991 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:58:13.992 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:58:13.992 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:58:13.992 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:58:13.992 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:58:13.992 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:13.992 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:13.992 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:58:13.992 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:58:13.993 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T21:58:14.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T21:58:14.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T21:58:14.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:58:14.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:58:14.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:58:14.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T21:58:14.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:58:14.158 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:58:14.158 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:58:14.158 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:58:14.158 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:58:14.158 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:14.158 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:14.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:58:14.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:58:14.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:58:14.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T21:58:14.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T21:58:14.282 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T21:58:14.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:58:15.293 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T21:58:15.293 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:58:15.293 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:58:15.293 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:58:15.293 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:15.293 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:15.293 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:58:15.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T21:58:15.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T21:58:15.342 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:58:15.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T21:58:15.349 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:58:13.923+0000 7f62b0beeec0 0 load: jerasure load: lrc 2026-02-22T21:58:15.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2026-02-22T21:58:15.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2026-02-22T21:58:15.351 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:452: TEST_ec_recovery_multiple_objects: local objname=myobject 2026-02-22T21:58:15.351 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:58:15.351 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:455: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:58:15.351 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_objects: setup_osds 7 2026-02-22T21:58:15.351 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2026-02-22T21:58:15.351 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T21:58:15.351 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2026-02-22T21:58:15.351 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2026-02-22T21:58:15.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:58:15.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T21:58:15.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:58:15.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:58:15.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:58:15.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:58:15.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T21:58:15.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:58:15.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:58:15.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:58:15.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:58:15.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T21:58:15.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T21:58:15.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:58:15.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:58:15.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:58:15.357 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:58:15.357 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:58:15.357 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:58:15.357 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:58:15.357 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:15.357 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:15.357 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:58:15.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:58:15.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:58:15.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:58:15.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:58:15.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:58:15.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:58:15.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:58:15.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:58:15.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:58:15.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:58:15.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:58:15.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:58:15.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T21:58:15.359 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:58:15.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1d3a0768-8612-4cdb-b791-0206ce72de92 2026-02-22T21:58:15.359 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 1d3a0768-8612-4cdb-b791-0206ce72de92 2026-02-22T21:58:15.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1d3a0768-8612-4cdb-b791-0206ce72de92' 2026-02-22T21:58:15.360 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:58:15.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3e5tphAQbFhAAcTNkY2U157KGt4ZfybR7wg== 2026-02-22T21:58:15.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3e5tphAQbFhAAcTNkY2U157KGt4ZfybR7wg=="}' 2026-02-22T21:58:15.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1d3a0768-8612-4cdb-b791-0206ce72de92 -i td/test-erasure-eio/0/new.json 2026-02-22T21:58:15.482 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:58:15.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T21:58:15.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3e5tphAQbFhAAcTNkY2U157KGt4ZfybR7wg== --osd-uuid 1d3a0768-8612-4cdb-b791-0206ce72de92 2026-02-22T21:58:15.508 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:15.507+0000 7fd721a7f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:15.510 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:15.511+0000 7fd721a7f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:15.511 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:15.511+0000 7fd721a7f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:15.511 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:15.511+0000 7fd721a7f8c0 -1 bdev(0x55cee504bc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:58:15.511 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:15.511+0000 7fd721a7f8c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T21:58:18.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T21:58:18.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:58:18.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:58:18.052 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:58:18.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:58:18.323 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:58:18.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:58:18.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:58:18.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:58:18.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:58:18.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:58:18.348 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:18.347+0000 7ff0a55538c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:18.350 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:18.351+0000 7ff0a55538c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:18.351 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:18.351+0000 7ff0a55538c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:18.526 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:58:18.526 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:58:18.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:58:18.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:58:18.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:58:18.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:18.527 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:58:18.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:58:18.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:18.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:58:18.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:19.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:19.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:19.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:58:19.722 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:58:19.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:19.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:58:19.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:20.084 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:20.083+0000 7ff0a55538c0 -1 Falling back to public interface 2026-02-22T21:58:20.916 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:20.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:20.917 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:58:20.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:58:20.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:20.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:58:21.113 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:22.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:22.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:22.115 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:58:22.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:58:22.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:22.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:58:22.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:23.077 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:23.075+0000 7ff0a55538c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:58:23.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:23.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:23.314 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:58:23.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:58:23.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:23.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:58:23.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:24.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:24.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:24.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:58:24.513 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:58:24.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:24.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:58:24.705 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/932801690,v1:127.0.0.1:6803/932801690] [v2:127.0.0.1:6804/932801690,v1:127.0.0.1:6805/932801690] exists,up 1d3a0768-8612-4cdb-b791-0206ce72de92 2026-02-22T21:58:24.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:58:24.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:58:24.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:58:24.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:58:24.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T21:58:24.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:58:24.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:58:24.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T21:58:24.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:58:24.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T21:58:24.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:58:24.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:58:24.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:58:24.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:58:24.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T21:58:24.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:58:24.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:58:24.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:58:24.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:58:24.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:58:24.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:58:24.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:58:24.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:58:24.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:58:24.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:58:24.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T21:58:24.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:58:24.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=25269b90-833c-4111-afd7-b34728e7ca85 2026-02-22T21:58:24.713 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 25269b90-833c-4111-afd7-b34728e7ca85 2026-02-22T21:58:24.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 25269b90-833c-4111-afd7-b34728e7ca85' 2026-02-22T21:58:24.713 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:58:24.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAAfJtpgE0iKxAAKt1UWGF/XiQLUx1lp1YPSQ== 2026-02-22T21:58:24.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAAfJtpgE0iKxAAKt1UWGF/XiQLUx1lp1YPSQ=="}' 2026-02-22T21:58:24.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 25269b90-833c-4111-afd7-b34728e7ca85 -i td/test-erasure-eio/1/new.json 2026-02-22T21:58:24.921 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:58:24.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T21:58:24.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAAfJtpgE0iKxAAKt1UWGF/XiQLUx1lp1YPSQ== --osd-uuid 25269b90-833c-4111-afd7-b34728e7ca85 2026-02-22T21:58:24.948 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:24.947+0000 7fc7043598c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:24.949 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:24.951+0000 7fc7043598c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:24.950 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:24.951+0000 7fc7043598c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:24.951 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:24.951+0000 7fc7043598c0 -1 bdev(0x557031f99c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:58:24.951 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:24.951+0000 7fc7043598c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T21:58:27.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T21:58:27.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:58:27.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:58:27.236 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T21:58:27.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:58:27.503 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T21:58:27.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T21:58:27.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:58:27.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:58:27.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:58:27.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:58:27.528 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:27.527+0000 7f66c76b68c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:27.530 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:27.531+0000 7f66c76b68c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:27.531 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:27.531+0000 7f66c76b68c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:27.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T21:58:27.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:58:27.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T21:58:27.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:58:27.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:58:27.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:27.705 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:58:27.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:58:27.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:27.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:58:27.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:28.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:28.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:28.902 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:58:28.903 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:58:28.903 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:28.903 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:58:29.015 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:29.015+0000 7f66c76b68c0 -1 Falling back to public interface 2026-02-22T21:58:29.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:30.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:30.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:30.100 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:58:30.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:58:30.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:30.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:58:30.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:31.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:31.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:31.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:58:31.298 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:58:31.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:31.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:58:31.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:32.265 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:32.263+0000 7f66c76b68c0 -1 osd.1 0 log_to_monitors true 2026-02-22T21:58:32.495 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:32.495 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:32.496 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:58:32.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:58:32.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:32.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:58:32.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:33.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:33.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:33.698 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:58:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:58:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:58:33.894 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2790262011,v1:127.0.0.1:6811/2790262011] [v2:127.0.0.1:6812/2790262011,v1:127.0.0.1:6813/2790262011] exists,up 25269b90-833c-4111-afd7-b34728e7ca85 2026-02-22T21:58:33.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:58:33.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:58:33.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:58:33.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:58:33.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T21:58:33.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:58:33.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:58:33.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T21:58:33.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:58:33.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T21:58:33.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:58:33.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:58:33.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:58:33.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:58:33.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T21:58:33.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:58:33.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:58:33.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:58:33.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:58:33.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:58:33.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:58:33.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:58:33.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:58:33.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:58:33.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:58:33.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T21:58:33.900 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:58:33.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5b059731-6127-497a-9752-e1f9c02138a1 2026-02-22T21:58:33.901 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 5b059731-6127-497a-9752-e1f9c02138a1 2026-02-22T21:58:33.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5b059731-6127-497a-9752-e1f9c02138a1' 2026-02-22T21:58:33.902 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:58:33.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJfJtpCrx3NhAAA/3+Cpd8AW7Kv6OZzxEu4w== 2026-02-22T21:58:33.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJfJtpCrx3NhAAA/3+Cpd8AW7Kv6OZzxEu4w=="}' 2026-02-22T21:58:33.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5b059731-6127-497a-9752-e1f9c02138a1 -i td/test-erasure-eio/2/new.json 2026-02-22T21:58:34.114 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:58:34.123 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T21:58:34.124 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJfJtpCrx3NhAAA/3+Cpd8AW7Kv6OZzxEu4w== --osd-uuid 5b059731-6127-497a-9752-e1f9c02138a1 2026-02-22T21:58:34.140 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:34.139+0000 7f5d2288e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:34.142 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:34.143+0000 7f5d2288e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:34.143 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:34.143+0000 7f5d2288e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:34.144 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:34.143+0000 7f5d2288e8c0 -1 bdev(0x558413ee3c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:58:34.144 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:34.143+0000 7f5d2288e8c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T21:58:36.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T21:58:36.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:58:36.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:58:36.408 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T21:58:36.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:58:36.681 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T21:58:36.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T21:58:36.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:58:36.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:58:36.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:58:36.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:58:36.707 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:36.707+0000 7fee9c36a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:36.708 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:36.707+0000 7fee9c36a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:36.709 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:36.707+0000 7fee9c36a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:36.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T21:58:36.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:58:36.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T21:58:36.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:58:36.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:58:36.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:36.878 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:58:36.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:58:36.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:36.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:58:37.078 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:38.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:38.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:38.080 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:58:38.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:58:38.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:38.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:58:38.247 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:38.247+0000 7fee9c36a8c0 -1 Falling back to public interface 2026-02-22T21:58:38.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:39.276 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:58:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:58:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:39.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:58:39.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:40.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:40.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:40.472 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:58:40.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:58:40.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:40.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:58:40.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:41.232 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:41.231+0000 7fee9c36a8c0 -1 osd.2 0 log_to_monitors true 2026-02-22T21:58:41.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:41.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:41.669 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:58:41.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:58:41.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:41.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:58:41.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:42.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:42.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:42.867 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:58:42.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:58:42.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:42.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:58:43.061 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2440585760,v1:127.0.0.1:6819/2440585760] [v2:127.0.0.1:6820/2440585760,v1:127.0.0.1:6821/2440585760] exists,up 5b059731-6127-497a-9752-e1f9c02138a1 2026-02-22T21:58:43.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:58:43.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:58:43.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:58:43.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:58:43.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T21:58:43.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:58:43.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:58:43.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T21:58:43.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:58:43.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T21:58:43.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:58:43.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:58:43.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:58:43.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:58:43.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T21:58:43.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T21:58:43.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:58:43.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:58:43.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:58:43.069 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:58:43.069 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:58:43.069 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:58:43.069 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:58:43.069 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:43.069 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:43.069 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:58:43.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:58:43.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:58:43.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:58:43.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:58:43.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:58:43.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:58:43.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:58:43.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:58:43.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:58:43.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:58:43.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:58:43.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:58:43.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T21:58:43.071 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:58:43.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2493a722-a260-46a3-b89e-56d1a6a3ad38 2026-02-22T21:58:43.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2493a722-a260-46a3-b89e-56d1a6a3ad38' 2026-02-22T21:58:43.071 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 2493a722-a260-46a3-b89e-56d1a6a3ad38 2026-02-22T21:58:43.071 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:58:43.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQATfJtp1IreBBAAK/idbD3SnUdbxUQWxvsJjQ== 2026-02-22T21:58:43.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQATfJtp1IreBBAAK/idbD3SnUdbxUQWxvsJjQ=="}' 2026-02-22T21:58:43.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2493a722-a260-46a3-b89e-56d1a6a3ad38 -i td/test-erasure-eio/3/new.json 2026-02-22T21:58:43.287 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:58:43.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T21:58:43.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATfJtp1IreBBAAK/idbD3SnUdbxUQWxvsJjQ== --osd-uuid 2493a722-a260-46a3-b89e-56d1a6a3ad38 2026-02-22T21:58:43.313 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:43.311+0000 7f444b89a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:43.314 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:43.315+0000 7f444b89a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:43.315 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:43.315+0000 7f444b89a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:43.316 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:43.315+0000 7f444b89a8c0 -1 bdev(0x563b90e01c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T21:58:43.316 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:43.315+0000 7f444b89a8c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T21:58:45.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T21:58:45.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:58:45.584 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T21:58:45.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T21:58:45.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:58:45.851 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T21:58:45.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T21:58:45.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:58:45.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:58:45.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:58:45.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:58:45.876 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:45.875+0000 7f34643c78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:45.878 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:45.879+0000 7f34643c78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:45.879 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:45.879+0000 7f34643c78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:46.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T21:58:46.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:58:46.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T21:58:46.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:58:46.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:58:46.059 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:46.059 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:58:46.059 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:58:46.059 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:46.060 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:58:46.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:47.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:47.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:47.258 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:58:47.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:58:47.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:47.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:58:47.399 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:47.398+0000 7f34643c78c0 -1 Falling back to public interface 2026-02-22T21:58:47.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:48.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:48.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:48.454 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:58:48.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:58:48.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:48.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:58:48.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:49.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:49.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:49.656 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:58:49.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:58:49.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:49.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:58:49.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:50.676 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:50.674+0000 7f34643c78c0 -1 osd.3 0 log_to_monitors true 2026-02-22T21:58:50.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:50.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:50.851 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:58:50.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:58:50.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:50.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:58:51.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:52.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:52.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:52.065 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:58:52.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:58:52.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:52.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T21:58:52.261 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 27 up_thru 28 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2907010509,v1:127.0.0.1:6827/2907010509] [v2:127.0.0.1:6828/2907010509,v1:127.0.0.1:6829/2907010509] exists,up 2493a722-a260-46a3-b89e-56d1a6a3ad38 2026-02-22T21:58:52.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:58:52.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:58:52.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:58:52.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:58:52.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-22T21:58:52.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:58:52.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:58:52.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2026-02-22T21:58:52.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:58:52.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-22T21:58:52.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:58:52.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:58:52.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:58:52.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:58:52.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-22T21:58:52.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-22T21:58:52.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:58:52.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:58:52.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:58:52.266 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:58:52.266 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:58:52.266 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:58:52.266 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:58:52.266 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:58:52.266 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:58:52.266 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:58:52.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:58:52.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:58:52.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:58:52.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:58:52.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:58:52.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:58:52.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:58:52.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:58:52.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:58:52.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:58:52.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:58:52.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:58:52.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-22T21:58:52.268 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:58:52.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bf978997-cb16-40bd-be96-2bb0831db226 2026-02-22T21:58:52.269 INFO:tasks.workunit.client.0.trial034.stdout:add osd4 bf978997-cb16-40bd-be96-2bb0831db226 2026-02-22T21:58:52.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 bf978997-cb16-40bd-be96-2bb0831db226' 2026-02-22T21:58:52.269 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:58:52.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAcfJtpMeqvEBAAqNHjOZat5eJKGJOwHR+0mA== 2026-02-22T21:58:52.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAcfJtpMeqvEBAAqNHjOZat5eJKGJOwHR+0mA=="}' 2026-02-22T21:58:52.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bf978997-cb16-40bd-be96-2bb0831db226 -i td/test-erasure-eio/4/new.json 2026-02-22T21:58:52.475 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:58:52.484 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-22T21:58:52.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAcfJtpMeqvEBAAqNHjOZat5eJKGJOwHR+0mA== --osd-uuid bf978997-cb16-40bd-be96-2bb0831db226 2026-02-22T21:58:52.502 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:52.502+0000 7fa00574e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:52.504 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:52.502+0000 7fa00574e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:52.505 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:52.502+0000 7fa00574e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:52.505 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:52.502+0000 7fa00574e8c0 -1 bdev(0x55ac05f0fc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-22T21:58:52.505 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:52.502+0000 7fa00574e8c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-22T21:58:54.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-22T21:58:54.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:58:54.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2026-02-22T21:58:54.768 INFO:tasks.workunit.client.0.trial034.stdout:adding osd4 key to auth repository 2026-02-22T21:58:54.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:58:55.033 INFO:tasks.workunit.client.0.trial034.stdout:start osd.4 2026-02-22T21:58:55.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2026-02-22T21:58:55.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:58:55.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:58:55.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:58:55.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:58:55.060 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:55.058+0000 7f5a0185b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:55.061 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:55.058+0000 7f5a0185b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:55.063 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:55.062+0000 7f5a0185b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:58:55.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2026-02-22T21:58:55.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:58:55.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2026-02-22T21:58:55.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:58:55.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:58:55.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:55.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:58:55.236 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:58:55.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:55.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:58:55.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:56.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:56.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:56.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:58:56.444 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:58:56.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:56.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:58:56.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:56.759 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:56.758+0000 7f5a0185b8c0 -1 Falling back to public interface 2026-02-22T21:58:57.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:57.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:57.651 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:58:57.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:58:57.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:57.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:58:57.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:58.845 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:58:58.845 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:58:58.846 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:58:58.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:58:58.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:58:58.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:58:59.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:58:59.719 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:58:59.718+0000 7f5a0185b8c0 -1 osd.4 0 log_to_monitors true 2026-02-22T21:59:00.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:00.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:00.055 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:59:00.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:59:00.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:00.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:59:00.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:01.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:01.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:01.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:59:01.264 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:59:01.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:01.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T21:59:01.462 INFO:tasks.workunit.client.0.trial034.stdout:osd.4 up in weight 1 up_from 36 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/630633658,v1:127.0.0.1:6835/630633658] [v2:127.0.0.1:6836/630633658,v1:127.0.0.1:6837/630633658] exists,up bf978997-cb16-40bd-be96-2bb0831db226 2026-02-22T21:59:01.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:59:01.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:59:01.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:59:01.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:59:01.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-22T21:59:01.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:59:01.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:59:01.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2026-02-22T21:59:01.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:59:01.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-22T21:59:01.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:59:01.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:59:01.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:59:01.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:59:01.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-22T21:59:01.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-22T21:59:01.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:59:01.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:59:01.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:59:01.467 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:59:01.467 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:59:01.467 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:59:01.467 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:59:01.467 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:01.467 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:01.467 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:59:01.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-22T21:59:01.469 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:59:01.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=13a2e9a3-f69a-4bc8-b569-705b078d9c57 2026-02-22T21:59:01.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 13a2e9a3-f69a-4bc8-b569-705b078d9c57' 2026-02-22T21:59:01.470 INFO:tasks.workunit.client.0.trial034.stdout:add osd5 13a2e9a3-f69a-4bc8-b569-705b078d9c57 2026-02-22T21:59:01.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:59:01.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlfJtprGq/HBAApvxtkdOZ1UDhcl/BlsdvGw== 2026-02-22T21:59:01.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlfJtprGq/HBAApvxtkdOZ1UDhcl/BlsdvGw=="}' 2026-02-22T21:59:01.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 13a2e9a3-f69a-4bc8-b569-705b078d9c57 -i td/test-erasure-eio/5/new.json 2026-02-22T21:59:01.685 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:59:01.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-22T21:59:01.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAlfJtprGq/HBAApvxtkdOZ1UDhcl/BlsdvGw== --osd-uuid 13a2e9a3-f69a-4bc8-b569-705b078d9c57 2026-02-22T21:59:01.712 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:01.710+0000 7f1c8b76d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:01.713 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:01.710+0000 7f1c8b76d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:01.714 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:01.714+0000 7f1c8b76d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:01.715 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:01.714+0000 7f1c8b76d8c0 -1 bdev(0x55ae836c1c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-22T21:59:01.715 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:01.714+0000 7f1c8b76d8c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-22T21:59:03.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-22T21:59:03.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:59:04.000 INFO:tasks.workunit.client.0.trial034.stdout:adding osd5 key to auth repository 2026-02-22T21:59:04.000 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2026-02-22T21:59:04.000 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:59:04.268 INFO:tasks.workunit.client.0.trial034.stdout:start osd.5 2026-02-22T21:59:04.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2026-02-22T21:59:04.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:59:04.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:59:04.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:59:04.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:59:04.284 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:04.282+0000 7fd0adc678c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:04.286 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:04.286+0000 7fd0adc678c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:04.287 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:04.286+0000 7fd0adc678c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:04.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2026-02-22T21:59:04.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:59:04.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2026-02-22T21:59:04.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:59:04.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:59:04.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:04.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:59:04.469 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:59:04.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:04.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:59:04.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:05.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:05.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:05.667 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:59:05.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:59:05.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:05.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:59:05.819 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:05.818+0000 7fd0adc678c0 -1 Falling back to public interface 2026-02-22T21:59:05.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:06.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:06.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:06.863 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:59:06.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:59:06.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:06.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:59:07.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:08.060 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:08.060 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:08.061 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:59:08.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:59:08.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:08.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:59:08.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:09.073 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:09.070+0000 7fd0adc678c0 -1 osd.5 0 log_to_monitors true 2026-02-22T21:59:09.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:09.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:09.256 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:59:09.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:59:09.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:09.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:59:09.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:10.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:10.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:10.454 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:59:10.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:59:10.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:10.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T21:59:10.647 INFO:tasks.workunit.client.0.trial034.stdout:osd.5 up in weight 1 up_from 44 up_thru 45 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1746424124,v1:127.0.0.1:6843/1746424124] [v2:127.0.0.1:6844/1746424124,v1:127.0.0.1:6845/1746424124] exists,up 13a2e9a3-f69a-4bc8-b569-705b078d9c57 2026-02-22T21:59:10.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:59:10.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:59:10.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:59:10.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:59:10.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-22T21:59:10.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:59:10.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:59:10.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2026-02-22T21:59:10.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:59:10.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-22T21:59:10.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:59:10.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:59:10.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:59:10.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:59:10.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-22T21:59:10.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-22T21:59:10.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:59:10.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:59:10.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:59:10.652 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:59:10.652 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:59:10.652 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:59:10.652 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:59:10.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:59:10.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:59:10.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:59:10.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-22T21:59:10.654 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:59:10.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=960a1631-93f1-4171-a702-e5b0c29ee648 2026-02-22T21:59:10.655 INFO:tasks.workunit.client.0.trial034.stdout:add osd6 960a1631-93f1-4171-a702-e5b0c29ee648 2026-02-22T21:59:10.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 960a1631-93f1-4171-a702-e5b0c29ee648' 2026-02-22T21:59:10.655 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:59:10.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAufJtpZSO6JxAABaBRoqcn2sOK8H+NiNYEww== 2026-02-22T21:59:10.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAufJtpZSO6JxAABaBRoqcn2sOK8H+NiNYEww=="}' 2026-02-22T21:59:10.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 960a1631-93f1-4171-a702-e5b0c29ee648 -i td/test-erasure-eio/6/new.json 2026-02-22T21:59:10.871 INFO:tasks.workunit.client.0.trial034.stdout:6 2026-02-22T21:59:10.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-22T21:59:10.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAufJtpZSO6JxAABaBRoqcn2sOK8H+NiNYEww== --osd-uuid 960a1631-93f1-4171-a702-e5b0c29ee648 2026-02-22T21:59:10.899 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:10.898+0000 7f52fd6538c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:10.901 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:10.898+0000 7f52fd6538c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:10.902 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:10.902+0000 7f52fd6538c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:10.902 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:10.902+0000 7f52fd6538c0 -1 bdev(0x564b8675fc00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-22T21:59:10.902 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:10.902+0000 7f52fd6538c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-22T21:59:13.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-22T21:59:13.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:59:13.200 INFO:tasks.workunit.client.0.trial034.stdout:adding osd6 key to auth repository 2026-02-22T21:59:13.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2026-02-22T21:59:13.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:59:13.470 INFO:tasks.workunit.client.0.trial034.stdout:start osd.6 2026-02-22T21:59:13.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2026-02-22T21:59:13.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:59:13.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:59:13.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:59:13.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:59:13.495 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:13.494+0000 7f74a85808c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:13.497 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:13.494+0000 7f74a85808c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:13.498 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:13.498+0000 7f74a85808c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:13.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2026-02-22T21:59:13.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:59:13.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2026-02-22T21:59:13.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:59:13.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:59:13.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:13.675 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:59:13.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:59:13.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:13.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:59:13.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:14.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:14.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:14.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:59:14.876 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:59:14.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:14.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:59:15.015 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:15.014+0000 7f74a85808c0 -1 Falling back to public interface 2026-02-22T21:59:15.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:16.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:16.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:16.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:59:16.072 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:59:16.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:16.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:59:16.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:17.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:17.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:17.270 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:59:17.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:59:17.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:17.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:59:17.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:18.043 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:18.042+0000 7f74a85808c0 -1 osd.6 0 log_to_monitors true 2026-02-22T21:59:18.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:18.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:18.473 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:59:18.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:59:18.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:18.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T21:59:18.676 INFO:tasks.workunit.client.0.trial034.stdout:osd.6 up in weight 1 up_from 52 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3618616671,v1:127.0.0.1:6851/3618616671] [v2:127.0.0.1:6852/3618616671,v1:127.0.0.1:6853/3618616671] exists,up 960a1631-93f1-4171-a702-e5b0c29ee648 2026-02-22T21:59:18.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:59:18.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:59:18.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:59:18.677 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T21:59:18.677 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T21:59:18.677 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:59:18.678 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:59:18.679 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:18.679 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:18.679 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T21:59:18.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T21:59:18.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T21:59:18.734 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:59:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T21:59:18.742 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:58:20.087+0000 7ff0a55538c0 0 load: jerasure load: lrc 2026-02-22T21:59:18.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:59:18.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:459: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2026-02-22T21:59:18.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:460: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2026-02-22T21:59:18.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T21:59:18.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T21:59:18.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-22T21:59:18.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T21:59:18.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-22T21:59:18.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T21:59:18.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-22T21:59:18.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T21:59:18.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T21:59:19.291 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T21:59:19.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:59:20.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T21:59:20.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:59:20.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:59:20.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:59:20.304 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:59:20.304 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:59:20.305 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:59:20.305 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:59:20.305 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:59:20.305 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:59:20.411 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:59:20.411 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:59:20.411 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:59:20.411 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:59:20.411 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:59:20.411 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:59:20.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:59:20.605 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:59:20.605 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:59:20.605 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:59:20.606 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:59:20.606 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:59:20.606 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T21:59:20.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:59:20.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:20.606 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:59:20.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803790 2026-02-22T21:59:20.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803790 2026-02-22T21:59:20.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790' 2026-02-22T21:59:20.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:20.682 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:59:20.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607564 2026-02-22T21:59:20.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607564 2026-02-22T21:59:20.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564' 2026-02-22T21:59:20.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:20.757 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:59:20.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411338 2026-02-22T21:59:20.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411338 2026-02-22T21:59:20.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338' 2026-02-22T21:59:20.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:20.834 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:59:20.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=115964117000 2026-02-22T21:59:20.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 115964117000 2026-02-22T21:59:20.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338 3-115964117000' 2026-02-22T21:59:20.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:20.909 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:59:20.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=154618822663 2026-02-22T21:59:20.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 154618822663 2026-02-22T21:59:20.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338 3-115964117000 4-154618822663' 2026-02-22T21:59:20.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:20.983 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:59:21.060 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=188978561029 2026-02-22T21:59:21.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 188978561029 2026-02-22T21:59:21.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338 3-115964117000 4-154618822663 5-188978561029' 2026-02-22T21:59:21.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:21.061 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:59:21.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=223338299395 2026-02-22T21:59:21.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 223338299395 2026-02-22T21:59:21.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338 3-115964117000 4-154618822663 5-188978561029 6-223338299395' 2026-02-22T21:59:21.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:21.136 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803790 2026-02-22T21:59:21.136 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:21.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:59:21.137 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803790 2026-02-22T21:59:21.137 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:21.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803790 2026-02-22T21:59:21.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803790' 2026-02-22T21:59:21.139 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803790 2026-02-22T21:59:21.139 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:59:21.331 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803788 -lt 25769803790 2026-02-22T21:59:21.331 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:59:22.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:59:22.333 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:59:22.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803790 -lt 25769803790 2026-02-22T21:59:22.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:22.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607564 2026-02-22T21:59:22.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:22.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:59:22.538 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607564 2026-02-22T21:59:22.538 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:22.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607564 2026-02-22T21:59:22.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607564' 2026-02-22T21:59:22.540 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607564 2026-02-22T21:59:22.540 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:59:22.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607564 -lt 51539607564 2026-02-22T21:59:22.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:22.746 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411338 2026-02-22T21:59:22.746 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:22.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:59:22.748 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411338 2026-02-22T21:59:22.748 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:22.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411338 2026-02-22T21:59:22.750 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 77309411338 2026-02-22T21:59:22.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411338' 2026-02-22T21:59:22.750 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:59:22.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411338 -lt 77309411338 2026-02-22T21:59:22.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:22.945 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-115964117000 2026-02-22T21:59:22.945 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:22.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:59:22.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-115964117000 2026-02-22T21:59:22.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:22.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=115964117000 2026-02-22T21:59:22.948 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 115964117000 2026-02-22T21:59:22.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 115964117000' 2026-02-22T21:59:22.949 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:59:23.158 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 115964117001 -lt 115964117000 2026-02-22T21:59:23.158 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:23.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-154618822663 2026-02-22T21:59:23.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:23.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:59:23.161 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-154618822663 2026-02-22T21:59:23.161 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:23.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=154618822663 2026-02-22T21:59:23.162 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 154618822663 2026-02-22T21:59:23.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 154618822663' 2026-02-22T21:59:23.163 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:59:23.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 154618822663 -lt 154618822663 2026-02-22T21:59:23.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:23.362 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-188978561029 2026-02-22T21:59:23.362 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:23.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:59:23.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-188978561029 2026-02-22T21:59:23.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:23.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=188978561029 2026-02-22T21:59:23.365 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 188978561029 2026-02-22T21:59:23.365 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 188978561029' 2026-02-22T21:59:23.365 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:59:23.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 188978561029 -lt 188978561029 2026-02-22T21:59:23.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:23.573 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-223338299395 2026-02-22T21:59:23.573 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:23.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:59:23.575 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-223338299395 2026-02-22T21:59:23.575 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:23.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=223338299395 2026-02-22T21:59:23.576 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 223338299395 2026-02-22T21:59:23.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 223338299395' 2026-02-22T21:59:23.577 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:59:23.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 223338299395 -lt 223338299395 2026-02-22T21:59:23.783 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:59:23.783 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:59:23.783 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:59:24.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T21:59:24.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:59:24.041 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:59:24.041 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:59:24.041 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:59:24.041 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:59:24.041 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:59:24.041 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:59:24.238 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T21:59:24.239 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:59:24.239 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:59:24.239 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:59:24.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T21:59:24.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:59:24.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:59:24.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:462: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2026-02-22T21:59:24.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T21:59:24.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T21:59:24.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2026-02-22T21:59:24.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:59:24.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T21:59:24.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:59:24.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T21:59:24.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:59:24.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T21:59:24.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:59:24.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T21:59:24.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T21:59:24.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:464: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T21:59:24.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2026-02-22T21:59:24.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:466: TEST_ec_recovery_multiple_objects: ceph osd out 0 2026-02-22T21:59:24.809 INFO:tasks.workunit.client.0.trial034.stderr:osd.0 is already out. 2026-02-22T21:59:24.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:469: TEST_ec_recovery_multiple_objects: wait_for_clean 2026-02-22T21:59:24.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T21:59:24.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T21:59:24.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T21:59:24.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:59:24.821 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:59:24.821 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T21:59:24.821 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T21:59:24.821 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T21:59:24.821 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:59:24.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:59:24.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T21:59:24.927 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T21:59:24.927 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T21:59:24.927 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T21:59:24.927 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T21:59:25.128 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T21:59:25.128 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T21:59:25.128 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T21:59:25.129 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T21:59:25.129 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T21:59:25.129 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T21:59:25.129 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T21:59:25.129 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T21:59:25.129 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:25.129 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:59:25.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803793 2026-02-22T21:59:25.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803793 2026-02-22T21:59:25.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793' 2026-02-22T21:59:25.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:25.205 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:59:25.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607567 2026-02-22T21:59:25.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607567 2026-02-22T21:59:25.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567' 2026-02-22T21:59:25.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:25.281 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:59:25.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411341 2026-02-22T21:59:25.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411341 2026-02-22T21:59:25.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-77309411341' 2026-02-22T21:59:25.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:25.358 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T21:59:25.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=115964117003 2026-02-22T21:59:25.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 115964117003 2026-02-22T21:59:25.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-77309411341 3-115964117003' 2026-02-22T21:59:25.431 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:25.432 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T21:59:25.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=154618822665 2026-02-22T21:59:25.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 154618822665 2026-02-22T21:59:25.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-77309411341 3-115964117003 4-154618822665' 2026-02-22T21:59:25.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:25.509 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T21:59:25.583 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=188978561032 2026-02-22T21:59:25.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 188978561032 2026-02-22T21:59:25.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-77309411341 3-115964117003 4-154618822665 5-188978561032' 2026-02-22T21:59:25.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T21:59:25.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T21:59:25.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=223338299398 2026-02-22T21:59:25.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 223338299398 2026-02-22T21:59:25.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-77309411341 3-115964117003 4-154618822665 5-188978561032 6-223338299398' 2026-02-22T21:59:25.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:25.660 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803793 2026-02-22T21:59:25.660 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:25.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T21:59:25.662 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803793 2026-02-22T21:59:25.662 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:25.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803793 2026-02-22T21:59:25.664 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803793 2026-02-22T21:59:25.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803793' 2026-02-22T21:59:25.664 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:59:25.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803791 -lt 25769803793 2026-02-22T21:59:25.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T21:59:26.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:59:26.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:59:27.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803793 -lt 25769803793 2026-02-22T21:59:27.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:27.068 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607567 2026-02-22T21:59:27.068 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:27.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T21:59:27.070 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607567 2026-02-22T21:59:27.070 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:27.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607567 2026-02-22T21:59:27.072 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607567 2026-02-22T21:59:27.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607567' 2026-02-22T21:59:27.072 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:59:27.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607567 -lt 51539607567 2026-02-22T21:59:27.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:27.276 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411341 2026-02-22T21:59:27.277 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:27.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T21:59:27.278 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411341 2026-02-22T21:59:27.279 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:27.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411341 2026-02-22T21:59:27.280 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 77309411341 2026-02-22T21:59:27.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411341' 2026-02-22T21:59:27.281 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:59:27.483 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411341 -lt 77309411341 2026-02-22T21:59:27.484 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:27.484 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-115964117003 2026-02-22T21:59:27.484 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:27.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T21:59:27.486 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-115964117003 2026-02-22T21:59:27.486 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:27.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=115964117003 2026-02-22T21:59:27.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 115964117003' 2026-02-22T21:59:27.489 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 115964117003 2026-02-22T21:59:27.489 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T21:59:27.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 115964117003 -lt 115964117003 2026-02-22T21:59:27.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:27.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-154618822665 2026-02-22T21:59:27.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:27.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T21:59:27.699 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-154618822665 2026-02-22T21:59:27.699 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:27.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=154618822665 2026-02-22T21:59:27.701 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 154618822665 2026-02-22T21:59:27.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 154618822665' 2026-02-22T21:59:27.701 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T21:59:27.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 154618822666 -lt 154618822665 2026-02-22T21:59:27.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:27.912 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-188978561032 2026-02-22T21:59:27.912 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:27.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T21:59:27.913 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-188978561032 2026-02-22T21:59:27.914 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:27.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=188978561032 2026-02-22T21:59:27.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 188978561032' 2026-02-22T21:59:27.915 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 188978561032 2026-02-22T21:59:27.916 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T21:59:28.108 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 188978561032 -lt 188978561032 2026-02-22T21:59:28.108 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T21:59:28.108 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-223338299398 2026-02-22T21:59:28.109 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T21:59:28.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T21:59:28.110 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-223338299398 2026-02-22T21:59:28.110 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T21:59:28.112 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=223338299398 2026-02-22T21:59:28.112 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 223338299398 2026-02-22T21:59:28.113 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 223338299398' 2026-02-22T21:59:28.113 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T21:59:28.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 223338299398 -lt 223338299398 2026-02-22T21:59:28.319 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:59:28.320 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:59:28.320 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:59:28.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T21:59:28.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T21:59:28.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T21:59:28.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T21:59:28.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:59:28.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:59:28.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:59:28.585 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:59:28.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T21:59:28.778 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T21:59:28.779 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T21:59:28.779 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:59:29.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T21:59:29.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T21:59:29.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T21:59:29.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2026-02-22T21:59:29.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T21:59:29.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T21:59:29.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2026-02-22T21:59:29.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T21:59:29.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T21:59:29.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2026-02-22T21:59:29.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T21:59:29.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T21:59:29.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2026-02-22T21:59:29.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T21:59:29.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T21:59:29.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2026-02-22T21:59:29.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T21:59:29.057 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T21:59:29.057 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2026-02-22T21:59:29.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T21:59:29.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T21:59:29.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:473: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2026-02-22T21:59:29.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T21:59:29.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T21:59:29.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2026-02-22T21:59:29.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T21:59:29.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T21:59:29.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2026-02-22T21:59:29.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T21:59:29.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T21:59:29.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2026-02-22T21:59:29.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T21:59:29.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T21:59:29.350 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T21:59:29.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T21:59:29.618 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T21:59:29.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T21:59:29.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T21:59:29.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:59:29.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T21:59:29.628 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:59:29.628 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:59:29.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:59:29.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:59:29.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:59:29.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:59:29.943 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:59:29.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:59:29.944 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:59:29.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:59:29.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:59:29.947 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:59:29.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:59:29.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:59:29.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:59:29.949 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:59:29.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:59:29.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:59:29.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T21:59:29.974 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:59:29.974 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:29.974 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:29.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T21:59:29.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:59:29.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:59:29.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T21:59:29.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T21:59:29.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T21:59:29.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T21:59:29.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T21:59:29.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T21:59:29.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T21:59:29.977 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T21:59:29.977 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T21:59:29.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T21:59:29.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T21:59:29.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T21:59:29.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T21:59:29.979 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T21:59:29.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:59:29.980 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T21:59:29.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:59:29.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T21:59:29.981 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:59:29.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:59:29.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T21:59:29.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T21:59:29.983 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:59:29.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:59:29.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:59:29.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T21:59:29.986 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T21:59:29.987 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:29.987 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:29.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T21:59:29.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T21:59:29.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T21:59:29.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T21:59:29.989 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T21:59:29.989 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:29.989 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:29.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T21:59:29.990 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T21:59:29.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:59:29.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T21:59:29.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T21:59:29.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T21:59:29.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T21:59:29.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T21:59:29.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T21:59:29.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T21:59:29.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T21:59:29.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T21:59:30.013 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T21:59:30.014 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:59:30.014 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:59:30.014 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:59:30.014 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:30.014 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:30.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:30.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T21:59:30.042 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T21:59:30.043 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T21:59:30.043 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:59:30.043 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:59:30.043 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T21:59:30.045 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T21:59:30.045 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:59:30.045 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:59:30.045 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:59:30.045 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:59:30.045 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:30.045 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:30.045 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:59:30.045 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:59:30.045 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T21:59:30.101 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T21:59:30.101 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T21:59:30.101 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T21:59:30.101 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T21:59:30.101 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T21:59:30.102 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T21:59:30.103 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:59:30.103 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:59:30.103 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:59:30.103 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:30.103 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:30.103 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:59:30.103 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T21:59:30.103 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T21:59:30.158 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T21:59:30.159 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T21:59:30.159 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T21:59:30.159 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T21:59:30.159 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T21:59:30.159 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T21:59:30.159 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:59:30.268 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T21:59:30.268 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:59:30.268 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:59:30.269 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:59:30.269 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:30.269 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:30.269 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:30.270 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:59:30.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:59:30.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T21:59:30.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T21:59:30.394 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T21:59:30.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T21:59:31.404 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T21:59:31.404 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T21:59:31.404 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:59:31.404 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T21:59:31.404 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:31.405 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:31.405 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T21:59:31.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T21:59:31.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T21:59:31.453 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T21:59:31.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T21:59:31.460 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:59:30.029+0000 7fddc8877ec0 0 load: jerasure load: lrc 2026-02-22T21:59:31.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2026-02-22T21:59:31.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2026-02-22T21:59:31.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:481: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2026-02-22T21:59:31.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T21:59:31.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:59:31.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:485: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2026-02-22T21:59:31.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2026-02-22T21:59:31.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T21:59:31.464 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2026-02-22T21:59:31.464 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2026-02-22T21:59:31.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:59:31.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T21:59:31.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:59:31.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:59:31.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T21:59:31.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:59:31.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T21:59:31.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:59:31.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:59:31.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:59:31.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:59:31.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T21:59:31.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T21:59:31.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:59:31.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:59:31.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:59:31.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:59:31.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:59:31.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:59:31.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:59:31.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:59:31.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:59:31.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:59:31.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T21:59:31.471 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:59:31.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b6777c96-bb0f-4b36-9bbb-fa0d180cb060 2026-02-22T21:59:31.472 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 b6777c96-bb0f-4b36-9bbb-fa0d180cb060 2026-02-22T21:59:31.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b6777c96-bb0f-4b36-9bbb-fa0d180cb060' 2026-02-22T21:59:31.472 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:59:31.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDfJtpKoC0HBAAV5ettQDKhv2dVDFSAHPtbw== 2026-02-22T21:59:31.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDfJtpKoC0HBAAV5ettQDKhv2dVDFSAHPtbw=="}' 2026-02-22T21:59:31.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b6777c96-bb0f-4b36-9bbb-fa0d180cb060 -i td/test-erasure-eio/0/new.json 2026-02-22T21:59:31.595 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:59:31.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T21:59:31.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDfJtpKoC0HBAAV5ettQDKhv2dVDFSAHPtbw== --osd-uuid b6777c96-bb0f-4b36-9bbb-fa0d180cb060 2026-02-22T21:59:31.621 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:31.617+0000 7f9808ca48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:31.623 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:31.621+0000 7f9808ca48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:31.623 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:31.621+0000 7f9808ca48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:31.624 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:31.621+0000 7f9808ca48c0 -1 bdev(0x558ea3115c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:59:31.624 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:31.621+0000 7f9808ca48c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T21:59:34.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T21:59:34.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:59:34.209 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T21:59:34.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:59:34.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:59:34.470 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T21:59:34.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T21:59:34.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:59:34.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:59:34.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:59:34.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:59:34.496 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:34.493+0000 7f17fa96b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:34.497 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:34.493+0000 7f17fa96b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:34.498 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:34.497+0000 7f17fa96b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:34.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T21:59:34.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:59:34.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T21:59:34.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:59:34.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:59:34.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:34.675 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:59:34.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:59:34.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:34.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:59:34.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:35.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:35.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:35.871 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:59:35.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:59:35.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:35.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:59:36.003 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:36.001+0000 7f17fa96b8c0 -1 Falling back to public interface 2026-02-22T21:59:36.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:37.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:37.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:37.074 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:59:37.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:59:37.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:37.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:59:37.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:38.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:38.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:38.270 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:59:38.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:59:38.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:38.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:59:38.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:39.038 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:39.033+0000 7f17fa96b8c0 -1 osd.0 0 log_to_monitors true 2026-02-22T21:59:39.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:39.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:39.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:59:39.468 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:59:39.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:39.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:59:39.674 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3795135131,v1:127.0.0.1:6803/3795135131] [v2:127.0.0.1:6804/3795135131,v1:127.0.0.1:6805/3795135131] exists,up b6777c96-bb0f-4b36-9bbb-fa0d180cb060 2026-02-22T21:59:39.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:59:39.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:59:39.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:59:39.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:59:39.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T21:59:39.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:59:39.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:59:39.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T21:59:39.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:59:39.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T21:59:39.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:59:39.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:39.679 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:39.680 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:39.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:39.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:59:39.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:59:39.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:59:39.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:59:39.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:59:39.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:59:39.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:59:39.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:59:39.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:59:39.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:59:39.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:59:39.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T21:59:39.681 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:59:39.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2b5073a7-f8da-4121-bcf9-8d2c63f3459e 2026-02-22T21:59:39.681 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 2b5073a7-f8da-4121-bcf9-8d2c63f3459e 2026-02-22T21:59:39.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2b5073a7-f8da-4121-bcf9-8d2c63f3459e' 2026-02-22T21:59:39.682 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:59:39.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLfJtp3jBPKRAA3sGZZXx2Txe9kolSw1UkQQ== 2026-02-22T21:59:39.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLfJtp3jBPKRAA3sGZZXx2Txe9kolSw1UkQQ=="}' 2026-02-22T21:59:39.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2b5073a7-f8da-4121-bcf9-8d2c63f3459e -i td/test-erasure-eio/1/new.json 2026-02-22T21:59:39.892 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:59:39.903 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T21:59:39.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLfJtp3jBPKRAA3sGZZXx2Txe9kolSw1UkQQ== --osd-uuid 2b5073a7-f8da-4121-bcf9-8d2c63f3459e 2026-02-22T21:59:39.929 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:39.925+0000 7f2de83248c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:39.931 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:39.929+0000 7f2de83248c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:39.932 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:39.929+0000 7f2de83248c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:39.933 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:39.929+0000 7f2de83248c0 -1 bdev(0x5601978cbc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:59:39.933 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:39.929+0000 7f2de83248c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T21:59:42.239 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T21:59:42.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:59:42.241 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T21:59:42.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:59:42.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:59:42.528 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T21:59:42.529 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T21:59:42.529 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:59:42.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:59:42.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:59:42.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:59:42.553 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:42.549+0000 7fb897f828c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:42.555 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:42.553+0000 7fb897f828c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:42.556 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:42.553+0000 7fb897f828c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:42.735 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T21:59:42.735 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:59:42.735 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T21:59:42.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:59:42.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:59:42.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:42.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:59:42.736 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:59:42.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:42.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:59:42.938 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:43.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:43.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:43.939 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:59:43.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:59:43.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:43.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:59:44.031 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:44.029+0000 7fb897f828c0 -1 Falling back to public interface 2026-02-22T21:59:44.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:45.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:45.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:45.134 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:59:45.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:59:45.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:45.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:59:45.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:46.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:46.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:46.334 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:59:46.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:59:46.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:46.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:59:46.529 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:47.287 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:47.285+0000 7fb897f828c0 -1 osd.1 0 log_to_monitors true 2026-02-22T21:59:47.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:47.531 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:47.531 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:59:47.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:59:47.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:47.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:59:47.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:48.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:48.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:48.730 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T21:59:48.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T21:59:48.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:48.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:59:48.929 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3369505885,v1:127.0.0.1:6811/3369505885] [v2:127.0.0.1:6812/3369505885,v1:127.0.0.1:6813/3369505885] exists,up 2b5073a7-f8da-4121-bcf9-8d2c63f3459e 2026-02-22T21:59:48.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:59:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:59:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:59:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:59:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T21:59:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:59:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:59:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T21:59:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:59:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T21:59:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:59:48.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:48.934 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:59:48.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:59:48.936 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T21:59:48.936 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:59:48.936 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=110d9a48-e184-4f67-86c1-13854a40dcf2 2026-02-22T21:59:48.936 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 110d9a48-e184-4f67-86c1-13854a40dcf2 2026-02-22T21:59:48.937 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 110d9a48-e184-4f67-86c1-13854a40dcf2' 2026-02-22T21:59:48.937 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:59:48.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBUfJtpsSx0OBAAxffSVwv5fD7dXJROHxkdZg== 2026-02-22T21:59:48.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBUfJtpsSx0OBAAxffSVwv5fD7dXJROHxkdZg=="}' 2026-02-22T21:59:48.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 110d9a48-e184-4f67-86c1-13854a40dcf2 -i td/test-erasure-eio/2/new.json 2026-02-22T21:59:49.153 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:59:49.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T21:59:49.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBUfJtpsSx0OBAAxffSVwv5fD7dXJROHxkdZg== --osd-uuid 110d9a48-e184-4f67-86c1-13854a40dcf2 2026-02-22T21:59:49.180 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:49.177+0000 7fc1ae67b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:49.181 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:49.177+0000 7fc1ae67b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:49.183 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:49.181+0000 7fc1ae67b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:49.183 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:49.181+0000 7fc1ae67b8c0 -1 bdev(0x55df23bd1c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:59:49.183 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:49.181+0000 7fc1ae67b8c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T21:59:51.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T21:59:51.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:59:51.724 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:59:51.724 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T21:59:51.724 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:59:51.995 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T21:59:51.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T21:59:51.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:59:51.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T21:59:51.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T21:59:51.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T21:59:52.020 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:52.017+0000 7eff71bd28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:52.021 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:52.017+0000 7eff71bd28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:52.022 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:52.021+0000 7eff71bd28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:52.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T21:59:52.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T21:59:52.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T21:59:52.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T21:59:52.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T21:59:52.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:52.198 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T21:59:52.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T21:59:52.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:52.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:59:52.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:53.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:53.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:53.393 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T21:59:53.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T21:59:53.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:53.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:59:53.488 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:53.485+0000 7eff71bd28c0 -1 Falling back to public interface 2026-02-22T21:59:53.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:54.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:54.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:54.590 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T21:59:54.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T21:59:54.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:54.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:59:54.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:55.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:55.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:55.783 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:59:55.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T21:59:55.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:55.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:59:55.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T21:59:56.496 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:56.493+0000 7eff71bd28c0 -1 osd.2 0 log_to_monitors true 2026-02-22T21:59:56.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T21:59:56.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T21:59:56.990 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T21:59:56.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T21:59:56.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T21:59:56.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:59:57.184 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2241490477,v1:127.0.0.1:6819/2241490477] [v2:127.0.0.1:6820/2241490477,v1:127.0.0.1:6821/2241490477] exists,up 110d9a48-e184-4f67-86c1-13854a40dcf2 2026-02-22T21:59:57.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T21:59:57.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T21:59:57.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T21:59:57.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T21:59:57.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T21:59:57.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T21:59:57.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T21:59:57.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T21:59:57.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T21:59:57.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T21:59:57.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T21:59:57.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:59:57.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:59:57.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:59:57.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T21:59:57.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T21:59:57.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:59:57.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T21:59:57.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T21:59:57.189 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T21:59:57.189 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T21:59:57.189 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T21:59:57.189 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T21:59:57.189 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T21:59:57.189 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T21:59:57.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T21:59:57.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T21:59:57.191 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T21:59:57.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=78724fe2-48b1-411c-ad78-7729fa3c01b4 2026-02-22T21:59:57.192 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 78724fe2-48b1-411c-ad78-7729fa3c01b4 2026-02-22T21:59:57.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 78724fe2-48b1-411c-ad78-7729fa3c01b4' 2026-02-22T21:59:57.192 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:59:57.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdfJtpdIIjDBAAJyg6NUrZFzyC5z1OaEgoEQ== 2026-02-22T21:59:57.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdfJtpdIIjDBAAJyg6NUrZFzyC5z1OaEgoEQ=="}' 2026-02-22T21:59:57.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 78724fe2-48b1-411c-ad78-7729fa3c01b4 -i td/test-erasure-eio/3/new.json 2026-02-22T21:59:57.395 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T21:59:57.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T21:59:57.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdfJtpdIIjDBAAJyg6NUrZFzyC5z1OaEgoEQ== --osd-uuid 78724fe2-48b1-411c-ad78-7729fa3c01b4 2026-02-22T21:59:57.423 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:57.421+0000 7f11667ce8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:57.425 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:57.421+0000 7f11667ce8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:57.426 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:57.421+0000 7f11667ce8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:59:57.426 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:57.425+0000 7f11667ce8c0 -1 bdev(0x561d2db75c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T21:59:57.426 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T21:59:57.425+0000 7f11667ce8c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T21:59:59.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T21:59:59.744 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T21:59:59.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T21:59:59.745 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T21:59:59.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:00:00.012 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T22:00:00.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T22:00:00.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:00:00.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:00:00.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:00:00.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:00:00.037 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:00.033+0000 7f28467158c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:00.038 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:00.037+0000 7f28467158c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:00.039 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:00.037+0000 7f28467158c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:00.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T22:00:00.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:00:00.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:00:00.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:00:00.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:00:00.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:00.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:00:00.220 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:00:00.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:00.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:00:00.416 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:01.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:01.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:01.418 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:00:01.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:00:01.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:01.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:00:01.524 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:01.520+0000 7f28467158c0 -1 Falling back to public interface 2026-02-22T22:00:01.615 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:02.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:02.616 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:02.617 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:00:02.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:00:02.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:02.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:00:02.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:03.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:03.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:03.823 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:00:03.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:00:03.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:03.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:00:04.028 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:04.534 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:04.532+0000 7f28467158c0 -1 osd.3 0 log_to_monitors true 2026-02-22T22:00:05.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:05.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:05.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:00:05.030 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:00:05.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:05.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:00:05.225 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1643129219,v1:127.0.0.1:6827/1643129219] [v2:127.0.0.1:6828/1643129219,v1:127.0.0.1:6829/1643129219] exists,up 78724fe2-48b1-411c-ad78-7729fa3c01b4 2026-02-22T22:00:05.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:00:05.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:00:05.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:00:05.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:00:05.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-22T22:00:05.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:00:05.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:00:05.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2026-02-22T22:00:05.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:00:05.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-22T22:00:05.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T22:00:05.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:00:05.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:00:05.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:00:05.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-22T22:00:05.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-22T22:00:05.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:00:05.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:00:05.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:00:05.230 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:00:05.230 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:00:05.230 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:00:05.230 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:00:05.230 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:05.230 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:05.230 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:00:05.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:00:05.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:00:05.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:00:05.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:00:05.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:00:05.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:00:05.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:00:05.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:00:05.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:00:05.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:00:05.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:00:05.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:00:05.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-22T22:00:05.232 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:00:05.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c3718897-7a05-425d-a422-9ee58d85c63a 2026-02-22T22:00:05.232 INFO:tasks.workunit.client.0.trial034.stdout:add osd4 c3718897-7a05-425d-a422-9ee58d85c63a 2026-02-22T22:00:05.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 c3718897-7a05-425d-a422-9ee58d85c63a' 2026-02-22T22:00:05.233 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:00:05.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlfJtpCL2FDhAA1X1AsSJH85l8rhx1ip0QHg== 2026-02-22T22:00:05.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlfJtpCL2FDhAA1X1AsSJH85l8rhx1ip0QHg=="}' 2026-02-22T22:00:05.243 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c3718897-7a05-425d-a422-9ee58d85c63a -i td/test-erasure-eio/4/new.json 2026-02-22T22:00:05.436 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:00:05.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-22T22:00:05.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlfJtpCL2FDhAA1X1AsSJH85l8rhx1ip0QHg== --osd-uuid c3718897-7a05-425d-a422-9ee58d85c63a 2026-02-22T22:00:05.463 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:05.460+0000 7f1054e1b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:05.465 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:05.460+0000 7f1054e1b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:05.466 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:05.464+0000 7f1054e1b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:05.466 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:05.464+0000 7f1054e1b8c0 -1 bdev(0x555cac38fc00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-22T22:00:05.467 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:05.464+0000 7f1054e1b8c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-22T22:00:07.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-22T22:00:07.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:00:07.780 INFO:tasks.workunit.client.0.trial034.stdout:adding osd4 key to auth repository 2026-02-22T22:00:07.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2026-02-22T22:00:07.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:00:08.047 INFO:tasks.workunit.client.0.trial034.stdout:start osd.4 2026-02-22T22:00:08.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2026-02-22T22:00:08.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:00:08.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:00:08.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:00:08.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:00:08.072 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:08.068+0000 7f61211418c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:08.074 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:08.068+0000 7f61211418c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:08.075 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:08.072+0000 7f61211418c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:08.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2026-02-22T22:00:08.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:00:08.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2026-02-22T22:00:08.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:00:08.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:00:08.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:08.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:00:08.253 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:00:08.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:08.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:00:08.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:09.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:09.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:09.451 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:00:09.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:00:09.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:09.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:00:09.608 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:09.604+0000 7f61211418c0 -1 Falling back to public interface 2026-02-22T22:00:09.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:10.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:10.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:10.655 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:00:10.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:00:10.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:10.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:00:10.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:11.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:11.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:11.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:00:11.857 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:00:11.857 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:11.858 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:00:12.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:12.628 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:12.624+0000 7f61211418c0 -1 osd.4 0 log_to_monitors true 2026-02-22T22:00:13.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:13.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:13.054 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:00:13.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:00:13.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:13.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:00:13.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:14.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:14.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:14.253 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:00:14.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:00:14.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:14.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:00:14.450 INFO:tasks.workunit.client.0.trial034.stdout:osd.4 up in weight 1 up_from 33 up_thru 34 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/556398565,v1:127.0.0.1:6835/556398565] [v2:127.0.0.1:6836/556398565,v1:127.0.0.1:6837/556398565] exists,up c3718897-7a05-425d-a422-9ee58d85c63a 2026-02-22T22:00:14.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:00:14.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:00:14.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:00:14.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:00:14.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-22T22:00:14.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:00:14.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:00:14.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2026-02-22T22:00:14.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:00:14.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-22T22:00:14.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T22:00:14.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:00:14.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:00:14.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:00:14.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-22T22:00:14.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-22T22:00:14.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:00:14.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:00:14.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:00:14.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:00:14.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:00:14.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:00:14.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:00:14.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:00:14.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:00:14.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:00:14.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-22T22:00:14.457 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:00:14.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa058ab7-fce1-4fec-b412-8a987b63032d 2026-02-22T22:00:14.457 INFO:tasks.workunit.client.0.trial034.stdout:add osd5 aa058ab7-fce1-4fec-b412-8a987b63032d 2026-02-22T22:00:14.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 aa058ab7-fce1-4fec-b412-8a987b63032d' 2026-02-22T22:00:14.458 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:00:14.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBufJtptA34GxAAHahIDfQ0GVsvRAVSyXr2xw== 2026-02-22T22:00:14.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBufJtptA34GxAAHahIDfQ0GVsvRAVSyXr2xw=="}' 2026-02-22T22:00:14.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa058ab7-fce1-4fec-b412-8a987b63032d -i td/test-erasure-eio/5/new.json 2026-02-22T22:00:14.674 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:00:14.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-22T22:00:14.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBufJtptA34GxAAHahIDfQ0GVsvRAVSyXr2xw== --osd-uuid aa058ab7-fce1-4fec-b412-8a987b63032d 2026-02-22T22:00:14.710 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:14.704+0000 7f51b9ccd8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:14.711 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:14.708+0000 7f51b9ccd8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:14.713 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:14.708+0000 7f51b9ccd8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:14.713 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:14.708+0000 7f51b9ccd8c0 -1 bdev(0x55ce230afc00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-22T22:00:14.713 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:14.708+0000 7f51b9ccd8c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-22T22:00:16.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-22T22:00:16.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:00:16.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2026-02-22T22:00:16.981 INFO:tasks.workunit.client.0.trial034.stdout:adding osd5 key to auth repository 2026-02-22T22:00:16.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:00:17.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2026-02-22T22:00:17.246 INFO:tasks.workunit.client.0.trial034.stdout:start osd.5 2026-02-22T22:00:17.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:00:17.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:00:17.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:00:17.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:00:17.271 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:17.268+0000 7f79825858c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:17.272 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:17.268+0000 7f79825858c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:17.273 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:17.268+0000 7f79825858c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:17.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2026-02-22T22:00:17.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:00:17.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2026-02-22T22:00:17.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:00:17.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:00:17.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:17.447 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:00:17.447 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:00:17.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:17.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T22:00:17.647 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:18.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:18.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:18.649 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:00:18.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:00:18.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:18.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T22:00:18.841 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:19.000 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:18.996+0000 7f79825858c0 -1 Falling back to public interface 2026-02-22T22:00:19.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:19.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:19.843 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:00:19.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:00:19.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:19.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T22:00:20.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:21.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:21.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:21.040 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:00:21.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:00:21.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:21.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T22:00:21.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:22.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:22.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:22.236 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:00:22.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:00:22.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:22.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T22:00:22.237 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:22.232+0000 7f79825858c0 -1 osd.5 0 log_to_monitors true 2026-02-22T22:00:22.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:23.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:23.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:23.438 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:00:23.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:00:23.439 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:23.439 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T22:00:23.633 INFO:tasks.workunit.client.0.trial034.stdout:osd.5 up in weight 1 up_from 42 up_thru 44 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2756595716,v1:127.0.0.1:6843/2756595716] [v2:127.0.0.1:6844/2756595716,v1:127.0.0.1:6845/2756595716] exists,up aa058ab7-fce1-4fec-b412-8a987b63032d 2026-02-22T22:00:23.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:00:23.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:00:23.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:00:23.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:00:23.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-22T22:00:23.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:00:23.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:00:23.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2026-02-22T22:00:23.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:00:23.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-22T22:00:23.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T22:00:23.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:00:23.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:00:23.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:00:23.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-22T22:00:23.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-22T22:00:23.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:00:23.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:00:23.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:00:23.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:00:23.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:00:23.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:00:23.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:00:23.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:00:23.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:00:23.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:00:23.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-22T22:00:23.640 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:00:23.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89361750-5115-41ec-8707-fdf07ff955ca 2026-02-22T22:00:23.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 89361750-5115-41ec-8707-fdf07ff955ca' 2026-02-22T22:00:23.641 INFO:tasks.workunit.client.0.trial034.stdout:add osd6 89361750-5115-41ec-8707-fdf07ff955ca 2026-02-22T22:00:23.642 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:00:23.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3fJtpK9vXJhAAaQSxlJJVdPOaUp+A0qt2DQ== 2026-02-22T22:00:23.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3fJtpK9vXJhAAaQSxlJJVdPOaUp+A0qt2DQ=="}' 2026-02-22T22:00:23.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89361750-5115-41ec-8707-fdf07ff955ca -i td/test-erasure-eio/6/new.json 2026-02-22T22:00:23.845 INFO:tasks.workunit.client.0.trial034.stdout:6 2026-02-22T22:00:23.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-22T22:00:23.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3fJtpK9vXJhAAaQSxlJJVdPOaUp+A0qt2DQ== --osd-uuid 89361750-5115-41ec-8707-fdf07ff955ca 2026-02-22T22:00:23.873 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:23.868+0000 7fce9f84e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:23.874 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:23.872+0000 7fce9f84e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:23.875 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:23.872+0000 7fce9f84e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:23.876 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:23.872+0000 7fce9f84e8c0 -1 bdev(0x56119fb8bc00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-22T22:00:23.876 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:23.872+0000 7fce9f84e8c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-22T22:00:26.179 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-22T22:00:26.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:00:26.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2026-02-22T22:00:26.181 INFO:tasks.workunit.client.0.trial034.stdout:adding osd6 key to auth repository 2026-02-22T22:00:26.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:00:26.450 INFO:tasks.workunit.client.0.trial034.stdout:start osd.6 2026-02-22T22:00:26.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2026-02-22T22:00:26.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:00:26.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:00:26.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:00:26.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:00:26.476 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:26.472+0000 7f6c173af8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:26.477 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:26.472+0000 7f6c173af8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:26.479 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:26.476+0000 7f6c173af8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:26.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2026-02-22T22:00:26.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:00:26.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2026-02-22T22:00:26.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:00:26.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:00:26.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:26.649 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:00:26.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:00:26.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:26.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T22:00:26.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:27.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:27.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:27.848 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:00:27.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:00:27.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:27.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T22:00:28.020 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:28.020+0000 7f6c173af8c0 -1 Falling back to public interface 2026-02-22T22:00:28.043 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:29.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:29.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:29.045 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:00:29.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:00:29.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:29.046 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T22:00:29.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:30.243 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:30.243 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:30.244 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:00:30.244 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:00:30.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:30.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T22:00:30.439 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:31.297 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:31.296+0000 7f6c173af8c0 -1 osd.6 0 log_to_monitors true 2026-02-22T22:00:31.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:31.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:31.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:00:31.440 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:00:31.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:31.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T22:00:31.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:32.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:32.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:32.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:00:32.640 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:00:32.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:32.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T22:00:32.840 INFO:tasks.workunit.client.0.trial034.stdout:osd.6 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2138744077,v1:127.0.0.1:6851/2138744077] [v2:127.0.0.1:6852/2138744077,v1:127.0.0.1:6853/2138744077] exists,up 89361750-5115-41ec-8707-fdf07ff955ca 2026-02-22T22:00:32.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:00:32.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:00:32.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:00:32.842 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T22:00:32.842 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:00:32.842 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:00:32.842 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:00:32.843 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:32.843 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:32.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T22:00:32.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T22:00:32.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T22:00:32.894 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:00:32.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T22:00:32.903 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T21:59:36.005+0000 7f17fa96b8c0 0 load: jerasure load: lrc 2026-02-22T22:00:32.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:00:32.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:488: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2026-02-22T22:00:32.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2026-02-22T22:00:32.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:00:32.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T22:00:32.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-22T22:00:32.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T22:00:32.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-22T22:00:32.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T22:00:32.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-22T22:00:33.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T22:00:33.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T22:00:33.555 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T22:00:33.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:00:34.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T22:00:34.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:00:34.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:00:34.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:00:34.566 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:00:34.567 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:00:34.568 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:00:34.568 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:00:34.568 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:00:34.568 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:00:34.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:00:34.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:00:34.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:00:34.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:00:34.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:00:34.678 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:00:34.875 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:00:34.875 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:00:34.875 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:00:34.875 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:00:34.875 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:00:34.875 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T22:00:34.875 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T22:00:34.875 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:00:34.875 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:34.876 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:00:34.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803789 2026-02-22T22:00:34.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803789 2026-02-22T22:00:34.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789' 2026-02-22T22:00:34.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:34.950 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:00:35.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607564 2026-02-22T22:00:35.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607564 2026-02-22T22:00:35.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-51539607564' 2026-02-22T22:00:35.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:35.025 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:00:35.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=73014444042 2026-02-22T22:00:35.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 73014444042 2026-02-22T22:00:35.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-51539607564 2-73014444042' 2026-02-22T22:00:35.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:35.100 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:00:35.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=103079215112 2026-02-22T22:00:35.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 103079215112 2026-02-22T22:00:35.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-51539607564 2-73014444042 3-103079215112' 2026-02-22T22:00:35.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:35.175 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T22:00:35.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=141733920775 2026-02-22T22:00:35.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 141733920775 2026-02-22T22:00:35.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-51539607564 2-73014444042 3-103079215112 4-141733920775' 2026-02-22T22:00:35.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:35.252 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T22:00:35.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=180388626437 2026-02-22T22:00:35.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 180388626437 2026-02-22T22:00:35.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-51539607564 2-73014444042 3-103079215112 4-141733920775 5-180388626437' 2026-02-22T22:00:35.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:35.328 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T22:00:35.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=210453397507 2026-02-22T22:00:35.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 210453397507 2026-02-22T22:00:35.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-51539607564 2-73014444042 3-103079215112 4-141733920775 5-180388626437 6-210453397507' 2026-02-22T22:00:35.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:35.405 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803789 2026-02-22T22:00:35.405 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:35.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:00:35.407 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803789 2026-02-22T22:00:35.407 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:35.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803789 2026-02-22T22:00:35.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803789' 2026-02-22T22:00:35.409 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803789 2026-02-22T22:00:35.409 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:00:35.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803787 -lt 25769803789 2026-02-22T22:00:35.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:00:36.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:00:36.614 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:00:36.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803790 -lt 25769803789 2026-02-22T22:00:36.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:36.812 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607564 2026-02-22T22:00:36.812 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:36.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:00:36.814 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607564 2026-02-22T22:00:36.814 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:36.815 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607564 2026-02-22T22:00:36.816 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607564 2026-02-22T22:00:36.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607564' 2026-02-22T22:00:36.816 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:00:37.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607564 -lt 51539607564 2026-02-22T22:00:37.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:37.017 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-73014444042 2026-02-22T22:00:37.017 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:37.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:00:37.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-73014444042 2026-02-22T22:00:37.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:37.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=73014444042 2026-02-22T22:00:37.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 73014444042' 2026-02-22T22:00:37.021 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 73014444042 2026-02-22T22:00:37.021 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:00:37.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 73014444042 -lt 73014444042 2026-02-22T22:00:37.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:37.227 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-103079215112 2026-02-22T22:00:37.228 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:37.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:00:37.229 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-103079215112 2026-02-22T22:00:37.229 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:37.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=103079215112 2026-02-22T22:00:37.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 103079215112' 2026-02-22T22:00:37.231 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 103079215112 2026-02-22T22:00:37.232 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:00:37.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 103079215113 -lt 103079215112 2026-02-22T22:00:37.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:37.429 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-141733920775 2026-02-22T22:00:37.429 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:37.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T22:00:37.431 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-141733920775 2026-02-22T22:00:37.431 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:37.432 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=141733920775 2026-02-22T22:00:37.432 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 141733920775' 2026-02-22T22:00:37.432 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 141733920775 2026-02-22T22:00:37.433 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T22:00:37.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 141733920775 -lt 141733920775 2026-02-22T22:00:37.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:37.637 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-180388626437 2026-02-22T22:00:37.637 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:37.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T22:00:37.639 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-180388626437 2026-02-22T22:00:37.639 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:37.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=180388626437 2026-02-22T22:00:37.641 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 180388626437 2026-02-22T22:00:37.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 180388626437' 2026-02-22T22:00:37.641 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T22:00:37.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 180388626437 -lt 180388626437 2026-02-22T22:00:37.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:37.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-210453397507 2026-02-22T22:00:37.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:37.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T22:00:37.850 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-210453397507 2026-02-22T22:00:37.850 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:37.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=210453397507 2026-02-22T22:00:37.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 210453397507' 2026-02-22T22:00:37.852 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 210453397507 2026-02-22T22:00:37.852 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T22:00:38.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 210453397507 -lt 210453397507 2026-02-22T22:00:38.050 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:00:38.050 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:00:38.050 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:00:38.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:00:38.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:00:38.316 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:00:38.316 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:00:38.317 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:00:38.317 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:00:38.317 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:00:38.317 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:00:38.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:00:38.511 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:00:38.511 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:00:38.511 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:00:38.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:00:38.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:00:38.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:00:38.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2026-02-22T22:00:38.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:00:38.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:00:38.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2026-02-22T22:00:38.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:00:38.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:00:38.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:00:38.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:00:38.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:00:38.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:00:38.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:00:38.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:00:38.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2026-02-22T22:00:38.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2026-02-22T22:00:38.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:00:38.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:00:38.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2026-02-22T22:00:38.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:00:38.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:00:38.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:00:38.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:00:38.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:00:38.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:00:38.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:00:38.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:00:38.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2026-02-22T22:00:38.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2026-02-22T22:00:38.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:00:38.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:00:38.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2026-02-22T22:00:38.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:00:38.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:00:38.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:00:38.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:00:38.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:00:38.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:00:38.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:00:38.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:00:38.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2026-02-22T22:00:38.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:496: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2026-02-22T22:00:38.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T22:00:38.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T22:00:38.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T22:00:38.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T22:00:38.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T22:00:38.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T22:00:38.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=myobject 2026-02-22T22:00:38.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T22:00:38.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T22:00:38.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T22:00:38.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=0 2026-02-22T22:00:38.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T22:00:38.836 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure myobject 2026-02-22T22:00:38.836 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:00:38.836 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2026-02-22T22:00:38.836 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-22T22:00:38.837 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:00:39.029 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:00:39.029 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T22:00:39.029 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:00:39.029 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:00:39.029 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T22:00:39.029 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2026-02-22T22:00:39.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2026-02-22T22:00:39.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T22:00:39.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=3 2026-02-22T22:00:39.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T22:00:39.030 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/3/type 2026-02-22T22:00:39.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T22:00:39.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-22T22:00:39.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T22:00:39.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2026-02-22T22:00:39.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T22:00:39.032 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T22:00:39.032 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T22:00:39.032 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2026-02-22T22:00:39.032 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T22:00:39.032 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T22:00:39.032 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:00:39.033 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:39.033 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:39.033 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T22:00:39.033 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-22T22:00:39.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T22:00:39.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T22:00:39.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T22:00:39.095 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.3 2026-02-22T22:00:39.095 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T22:00:39.095 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T22:00:39.095 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:00:39.095 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:39.095 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:39.096 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T22:00:39.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T22:00:39.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2026-02-22T22:00:39.152 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:497: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2026-02-22T22:00:39.401 INFO:tasks.workunit.client.0.trial034.stderr:osd.0 is already out. 2026-02-22T22:00:39.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2026-02-22T22:00:39.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:00:39.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:00:39.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:00:39.410 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:00:39.411 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:00:39.411 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:00:39.411 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:00:39.411 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:00:39.412 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:00:39.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:00:39.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:00:39.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:00:39.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:00:39.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:00:39.519 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:00:39.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:00:39.722 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:00:39.722 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:00:39.722 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:00:39.722 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:00:39.723 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T22:00:39.723 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T22:00:39.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:00:39.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:39.723 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:00:39.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803792 2026-02-22T22:00:39.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803792 2026-02-22T22:00:39.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803792' 2026-02-22T22:00:39.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:39.797 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:00:39.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607567 2026-02-22T22:00:39.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607567 2026-02-22T22:00:39.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803792 1-51539607567' 2026-02-22T22:00:39.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:39.870 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:00:39.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=73014444045 2026-02-22T22:00:39.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 73014444045 2026-02-22T22:00:39.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803792 1-51539607567 2-73014444045' 2026-02-22T22:00:39.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:39.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:00:40.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=103079215115 2026-02-22T22:00:40.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 103079215115 2026-02-22T22:00:40.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803792 1-51539607567 2-73014444045 3-103079215115' 2026-02-22T22:00:40.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:40.020 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T22:00:40.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=141733920778 2026-02-22T22:00:40.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 141733920778 2026-02-22T22:00:40.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803792 1-51539607567 2-73014444045 3-103079215115 4-141733920778' 2026-02-22T22:00:40.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:40.096 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T22:00:40.170 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=180388626440 2026-02-22T22:00:40.170 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 180388626440 2026-02-22T22:00:40.170 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803792 1-51539607567 2-73014444045 3-103079215115 4-141733920778 5-180388626440' 2026-02-22T22:00:40.171 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:00:40.171 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T22:00:40.244 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=210453397510 2026-02-22T22:00:40.244 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 210453397510 2026-02-22T22:00:40.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803792 1-51539607567 2-73014444045 3-103079215115 4-141733920778 5-180388626440 6-210453397510' 2026-02-22T22:00:40.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:40.245 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803792 2026-02-22T22:00:40.245 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:40.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:00:40.247 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803792 2026-02-22T22:00:40.247 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:40.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803792 2026-02-22T22:00:40.249 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803792 2026-02-22T22:00:40.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803792' 2026-02-22T22:00:40.249 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:00:40.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803790 -lt 25769803792 2026-02-22T22:00:40.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:00:41.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:00:41.446 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:00:41.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803793 -lt 25769803792 2026-02-22T22:00:41.651 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:41.651 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607567 2026-02-22T22:00:41.651 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:41.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:00:41.653 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607567 2026-02-22T22:00:41.653 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:41.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607567 2026-02-22T22:00:41.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607567' 2026-02-22T22:00:41.655 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607567 2026-02-22T22:00:41.656 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:00:41.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607567 -lt 51539607567 2026-02-22T22:00:41.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:41.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-73014444045 2026-02-22T22:00:41.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:41.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:00:41.850 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-73014444045 2026-02-22T22:00:41.850 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:41.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=73014444045 2026-02-22T22:00:41.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 73014444045' 2026-02-22T22:00:41.852 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 73014444045 2026-02-22T22:00:41.852 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:00:42.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 73014444045 -lt 73014444045 2026-02-22T22:00:42.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:42.062 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-103079215115 2026-02-22T22:00:42.062 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:42.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:00:42.064 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-103079215115 2026-02-22T22:00:42.064 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:42.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=103079215115 2026-02-22T22:00:42.065 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 103079215115 2026-02-22T22:00:42.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 103079215115' 2026-02-22T22:00:42.066 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:00:42.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 103079215115 -lt 103079215115 2026-02-22T22:00:42.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:42.261 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-141733920778 2026-02-22T22:00:42.261 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:42.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T22:00:42.263 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-141733920778 2026-02-22T22:00:42.263 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:42.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=141733920778 2026-02-22T22:00:42.265 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 141733920778 2026-02-22T22:00:42.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 141733920778' 2026-02-22T22:00:42.265 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T22:00:42.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 141733920778 -lt 141733920778 2026-02-22T22:00:42.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:42.465 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-180388626440 2026-02-22T22:00:42.465 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:42.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T22:00:42.467 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-180388626440 2026-02-22T22:00:42.467 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:42.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=180388626440 2026-02-22T22:00:42.469 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 180388626440 2026-02-22T22:00:42.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 180388626440' 2026-02-22T22:00:42.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T22:00:42.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 180388626440 -lt 180388626440 2026-02-22T22:00:42.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:00:42.675 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-210453397510 2026-02-22T22:00:42.675 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:00:42.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T22:00:42.677 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-210453397510 2026-02-22T22:00:42.677 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:00:42.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=210453397510 2026-02-22T22:00:42.679 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 210453397510 2026-02-22T22:00:42.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 210453397510' 2026-02-22T22:00:42.679 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T22:00:42.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 210453397511 -lt 210453397510 2026-02-22T22:00:42.881 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:00:42.882 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:00:42.882 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:00:43.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:00:43.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:00:43.146 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:00:43.146 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:00:43.146 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:00:43.146 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:00:43.147 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:00:43.147 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:00:43.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:00:43.345 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:00:43.345 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:00:43.345 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:00:43.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:00:43.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:00:43.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:00:43.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:502: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2026-02-22T22:00:43.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:00:43.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:00:43.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2026-02-22T22:00:43.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:00:43.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:00:43.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2026-02-22T22:00:43.626 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:00:43.626 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:00:43.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2026-02-22T22:00:43.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:00:43.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:00:43.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2026-02-22T22:00:43.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:00:43.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:00:43.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2026-02-22T22:00:43.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:00:43.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:00:43.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2026-02-22T22:00:43.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:00:43.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:00:43.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2026-02-22T22:00:43.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:00:43.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:00:43.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2026-02-22T22:00:43.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:00:43.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:00:43.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2026-02-22T22:00:43.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:00:43.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T22:00:43.927 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T22:00:43.937 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T22:00:44.183 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T22:00:44.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T22:00:44.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:00:44.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:00:44.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:00:44.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:00:44.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:00:44.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:00:44.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:00:44.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:00:44.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:00:44.305 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:00:44.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:00:44.306 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:00:44.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:00:44.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:00:44.308 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:00:44.309 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:00:44.309 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:00:44.310 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:00:44.310 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:00:44.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:00:44.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:00:44.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:00:44.335 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:00:44.335 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:44.335 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:44.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:00:44.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:00:44.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:00:44.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T22:00:44.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T22:00:44.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T22:00:44.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T22:00:44.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:00:44.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:00:44.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:00:44.338 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:00:44.338 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:00:44.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:00:44.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:00:44.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:00:44.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:00:44.340 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:00:44.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:00:44.341 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:00:44.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:00:44.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:00:44.343 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:00:44.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:00:44.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:00:44.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:00:44.345 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:00:44.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:00:44.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:00:44.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:00:44.349 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:00:44.349 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:44.349 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:44.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:00:44.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:00:44.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:00:44.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T22:00:44.352 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:00:44.352 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:44.352 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:44.352 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T22:00:44.353 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:00:44.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:00:44.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:00:44.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T22:00:44.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T22:00:44.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T22:00:44.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:00:44.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:00:44.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:00:44.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T22:00:44.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T22:00:44.376 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:00:44.377 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:00:44.377 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:00:44.377 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:00:44.377 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:44.377 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:44.377 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:00:44.378 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:00:44.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:00:44.407 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:00:44.407 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:00:44.407 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:00:44.407 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:00:44.408 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:00:44.408 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:00:44.408 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:00:44.408 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:00:44.409 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:00:44.409 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:44.409 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:44.410 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:00:44.410 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:00:44.410 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T22:00:44.463 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:00:44.463 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:00:44.463 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:00:44.464 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:00:44.464 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:00:44.464 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:00:44.464 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:00:44.464 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:00:44.465 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:00:44.465 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:44.465 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:44.465 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:00:44.466 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:00:44.466 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T22:00:44.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T22:00:44.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T22:00:44.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:00:44.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:00:44.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:00:44.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T22:00:44.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:00:44.628 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:00:44.629 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:00:44.629 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:00:44.629 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:00:44.629 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:44.629 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:44.629 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:00:44.630 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:00:44.630 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:00:44.647 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T22:00:44.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:00:44.752 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T22:00:44.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:00:45.763 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T22:00:45.763 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:00:45.763 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:00:45.763 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:00:45.763 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:45.764 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:45.764 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:00:45.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T22:00:45.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T22:00:45.813 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:00:45.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T22:00:45.820 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:00:44.395+0000 7fcb066a5ec0 0 load: jerasure load: lrc 2026-02-22T22:00:45.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2026-02-22T22:00:45.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2026-02-22T22:00:45.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:610: TEST_ec_recovery_unfound: local objname=myobject 2026-02-22T22:00:45.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_recovery_unfound: local lastobj=100 2026-02-22T22:00:45.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_recovery_unfound: local testobj=obj75 2026-02-22T22:00:45.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:00:45.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:616: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2026-02-22T22:00:45.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-22T22:00:45.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_recovery_unfound: setup_osds 5 2026-02-22T22:00:45.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2026-02-22T22:00:45.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T22:00:45.823 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2026-02-22T22:00:45.826 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2026-02-22T22:00:45.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:00:45.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T22:00:45.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:00:45.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:00:45.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:00:45.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:00:45.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T22:00:45.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-22T22:00:45.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:00:45.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:00:45.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:00:45.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T22:00:45.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T22:00:45.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:00:45.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:00:45.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:00:45.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:00:45.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:00:45.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:00:45.828 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:00:45.828 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:45.829 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:45.829 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:00:45.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:00:45.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:00:45.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:00:45.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:00:45.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:00:45.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:00:45.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:00:45.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:00:45.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:00:45.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:00:45.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:00:45.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:00:45.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T22:00:45.831 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:00:45.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=91bab6c6-a9bc-4ae5-a6f6-cdcd640f3caf 2026-02-22T22:00:45.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 91bab6c6-a9bc-4ae5-a6f6-cdcd640f3caf' 2026-02-22T22:00:45.831 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 91bab6c6-a9bc-4ae5-a6f6-cdcd640f3caf 2026-02-22T22:00:45.832 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:00:45.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCNfJtpY76MMhAAfjuwWt1eaye/5wNPvSHuAA== 2026-02-22T22:00:45.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCNfJtpY76MMhAAfjuwWt1eaye/5wNPvSHuAA=="}' 2026-02-22T22:00:45.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 91bab6c6-a9bc-4ae5-a6f6-cdcd640f3caf -i td/test-erasure-eio/0/new.json 2026-02-22T22:00:45.959 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:00:45.966 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T22:00:45.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNfJtpY76MMhAAfjuwWt1eaye/5wNPvSHuAA== --osd-uuid 91bab6c6-a9bc-4ae5-a6f6-cdcd640f3caf 2026-02-22T22:00:45.986 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:45.983+0000 7ff8a28698c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:45.987 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:45.987+0000 7ff8a28698c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:45.988 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:45.987+0000 7ff8a28698c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:45.989 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:45.987+0000 7ff8a28698c0 -1 bdev(0x55f853497c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:00:45.989 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:45.987+0000 7ff8a28698c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T22:00:48.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T22:00:48.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:00:48.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:00:48.552 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T22:00:48.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:00:48.831 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T22:00:48.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:00:48.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:00:48.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:00:48.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:00:48.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:00:48.858 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:48.859+0000 7f1878c778c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:48.860 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:48.859+0000 7f1878c778c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:48.861 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:48.859+0000 7f1878c778c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:49.028 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:00:49.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:00:49.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:00:49.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:00:49.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:00:49.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:49.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:00:49.029 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:00:49.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:49.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:00:49.238 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:50.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:50.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:50.240 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:00:50.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:00:50.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:50.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:00:50.368 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:50.367+0000 7f1878c778c0 -1 Falling back to public interface 2026-02-22T22:00:50.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:51.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:51.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:51.444 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:00:51.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:00:51.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:51.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:00:51.644 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:52.646 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:52.646 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:52.646 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:00:52.647 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:00:52.647 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:52.647 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:00:52.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:53.382 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:53.379+0000 7f1878c778c0 -1 osd.0 0 log_to_monitors true 2026-02-22T22:00:53.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:53.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:53.848 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:00:53.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:00:53.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:53.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:00:54.047 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3428960649,v1:127.0.0.1:6803/3428960649] [v2:127.0.0.1:6804/3428960649,v1:127.0.0.1:6805/3428960649] exists,up 91bab6c6-a9bc-4ae5-a6f6-cdcd640f3caf 2026-02-22T22:00:54.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:00:54.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:00:54.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:00:54.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:00:54.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T22:00:54.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:00:54.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:00:54.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:00:54.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:00:54.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:00:54.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-22T22:00:54.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:00:54.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:00:54.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:00:54.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:00:54.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:00:54.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:00:54.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:00:54.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:00:54.054 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:00:54.054 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:00:54.054 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:00:54.054 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:00:54.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:00:54.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:00:54.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:00:54.056 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:00:54.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=45f406c3-2da4-473b-9371-ccce453e3e5b 2026-02-22T22:00:54.057 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 45f406c3-2da4-473b-9371-ccce453e3e5b' 2026-02-22T22:00:54.057 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 45f406c3-2da4-473b-9371-ccce453e3e5b 2026-02-22T22:00:54.057 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:00:54.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCWfJtpQQIBBBAALL7K6GJ1QWLWMqM9I2qVjw== 2026-02-22T22:00:54.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCWfJtpQQIBBBAALL7K6GJ1QWLWMqM9I2qVjw=="}' 2026-02-22T22:00:54.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 45f406c3-2da4-473b-9371-ccce453e3e5b -i td/test-erasure-eio/1/new.json 2026-02-22T22:00:54.280 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:00:54.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T22:00:54.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWfJtpQQIBBBAALL7K6GJ1QWLWMqM9I2qVjw== --osd-uuid 45f406c3-2da4-473b-9371-ccce453e3e5b 2026-02-22T22:00:54.310 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:54.311+0000 7fb3862698c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:54.311 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:54.311+0000 7fb3862698c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:54.312 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:54.311+0000 7fb3862698c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:54.313 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:54.311+0000 7fb3862698c0 -1 bdev(0x562c0120fc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:00:54.313 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:54.311+0000 7fb3862698c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T22:00:56.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T22:00:56.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:00:56.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:00:56.577 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T22:00:56.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:00:56.854 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T22:00:56.854 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:00:56.854 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:00:56.854 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:00:56.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:00:56.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:00:56.873 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:56.871+0000 7f80201338c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:56.875 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:56.875+0000 7f80201338c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:56.875 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:56.875+0000 7f80201338c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:00:57.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:00:57.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:00:57.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:00:57.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:00:57.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:00:57.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:57.062 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:00:57.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:00:57.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:57.063 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:00:57.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:58.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:58.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:58.265 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:00:58.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:00:58.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:58.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:00:58.380 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:00:58.379+0000 7f80201338c0 -1 Falling back to public interface 2026-02-22T22:00:58.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:00:59.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:00:59.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:00:59.474 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:00:59.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:00:59.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:00:59.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:00:59.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:00.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:00.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:00.680 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:01:00.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:01:00.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:00.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:01:00.886 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:01.616 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:01.615+0000 7f80201338c0 -1 osd.1 0 log_to_monitors true 2026-02-22T22:01:01.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:01.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:01.889 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:01:01.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:01:01.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:01.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:01:02.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:03.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:03.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:03.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:01:03.092 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:01:03.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:03.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:01:03.290 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1600474601,v1:127.0.0.1:6811/1600474601] [v2:127.0.0.1:6812/1600474601,v1:127.0.0.1:6813/1600474601] exists,up 45f406c3-2da4-473b-9371-ccce453e3e5b 2026-02-22T22:01:03.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:01:03.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:01:03.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:01:03.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:01:03.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T22:01:03.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:01:03.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:01:03.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:01:03.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:01:03.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:01:03.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-22T22:01:03.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:01:03.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:01:03.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:01:03.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:01:03.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:01:03.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:01:03.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:01:03.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:01:03.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:01:03.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:01:03.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:01:03.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:01:03.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:01:03.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:01:03.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:01:03.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:01:03.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:01:03.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bf4fa814-1924-4b4e-804a-17051ab5081e 2026-02-22T22:01:03.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bf4fa814-1924-4b4e-804a-17051ab5081e' 2026-02-22T22:01:03.298 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 bf4fa814-1924-4b4e-804a-17051ab5081e 2026-02-22T22:01:03.298 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:01:03.310 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCffJtpPPiMEhAAhqVyNebuEnbZjT/Q4RjunQ== 2026-02-22T22:01:03.310 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCffJtpPPiMEhAAhqVyNebuEnbZjT/Q4RjunQ=="}' 2026-02-22T22:01:03.310 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bf4fa814-1924-4b4e-804a-17051ab5081e -i td/test-erasure-eio/2/new.json 2026-02-22T22:01:03.518 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:01:03.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T22:01:03.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCffJtpPPiMEhAAhqVyNebuEnbZjT/Q4RjunQ== --osd-uuid bf4fa814-1924-4b4e-804a-17051ab5081e 2026-02-22T22:01:03.547 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:03.547+0000 7f379329b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:03.548 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:03.547+0000 7f379329b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:03.549 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:03.547+0000 7f379329b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:03.550 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:03.547+0000 7f379329b8c0 -1 bdev(0x562918df9c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:01:03.550 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:03.547+0000 7f379329b8c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T22:01:06.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T22:01:06.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:01:06.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:01:06.096 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T22:01:06.097 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:01:06.379 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T22:01:06.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:01:06.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:01:06.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:01:06.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:01:06.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:01:06.406 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:06.403+0000 7f7a818f78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:06.408 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:06.407+0000 7f7a818f78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:06.409 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:06.407+0000 7f7a818f78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:06.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:01:06.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:01:06.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:01:06.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:01:06.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:01:06.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:06.581 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:01:06.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:01:06.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:06.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:01:06.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:07.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:07.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:07.786 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:01:07.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:01:07.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:07.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:01:07.916 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:07.915+0000 7f7a818f78c0 -1 Falling back to public interface 2026-02-22T22:01:07.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:08.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:08.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:08.987 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:01:08.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:01:08.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:08.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:01:09.185 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:10.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:10.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:10.188 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:01:10.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:01:10.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:10.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:01:10.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:10.972 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:10.971+0000 7f7a818f78c0 -1 osd.2 0 log_to_monitors true 2026-02-22T22:01:11.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:11.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:11.391 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:01:11.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:01:11.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:11.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:01:11.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:12.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:12.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:12.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:01:12.600 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:01:12.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:12.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:01:12.797 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/481708764,v1:127.0.0.1:6819/481708764] [v2:127.0.0.1:6820/481708764,v1:127.0.0.1:6821/481708764] exists,up bf4fa814-1924-4b4e-804a-17051ab5081e 2026-02-22T22:01:12.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:01:12.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:01:12.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:01:12.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:01:12.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T22:01:12.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:01:12.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:01:12.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T22:01:12.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:01:12.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:01:12.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-22T22:01:12.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:01:12.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:01:12.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:01:12.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:01:12.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:01:12.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:01:12.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:01:12.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:01:12.802 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:01:12.802 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:01:12.803 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:01:12.803 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:01:12.803 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:01:12.803 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:01:12.803 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:01:12.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:01:12.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:01:12.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:01:12.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:01:12.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:01:12.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:01:12.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:01:12.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:01:12.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:01:12.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:01:12.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:01:12.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:01:12.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:01:12.804 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:01:12.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=271adbe0-86e7-4264-8742-2c3415251445 2026-02-22T22:01:12.805 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 271adbe0-86e7-4264-8742-2c3415251445 2026-02-22T22:01:12.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 271adbe0-86e7-4264-8742-2c3415251445' 2026-02-22T22:01:12.806 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:01:12.815 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCofJtpkQCqMBAAqqC6VzhJVbusg1gatVUEpw== 2026-02-22T22:01:12.815 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCofJtpkQCqMBAAqqC6VzhJVbusg1gatVUEpw=="}' 2026-02-22T22:01:12.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 271adbe0-86e7-4264-8742-2c3415251445 -i td/test-erasure-eio/3/new.json 2026-02-22T22:01:13.016 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:01:13.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T22:01:13.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCofJtpkQCqMBAAqqC6VzhJVbusg1gatVUEpw== --osd-uuid 271adbe0-86e7-4264-8742-2c3415251445 2026-02-22T22:01:13.045 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:13.043+0000 7f33771c88c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:13.047 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:13.047+0000 7f33771c88c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:13.048 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:13.047+0000 7f33771c88c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:13.048 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:13.047+0000 7f33771c88c0 -1 bdev(0x56227edc7c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T22:01:13.048 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:13.047+0000 7f33771c88c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T22:01:15.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T22:01:15.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:01:15.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T22:01:15.328 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T22:01:15.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:01:15.602 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T22:01:15.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T22:01:15.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:01:15.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:01:15.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:01:15.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:01:15.630 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:15.631+0000 7f737cf478c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:15.632 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:15.631+0000 7f737cf478c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:15.633 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:15.631+0000 7f737cf478c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:15.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T22:01:15.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:01:15.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:01:15.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:01:15.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:01:15.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:15.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:01:15.801 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:01:15.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:15.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:01:16.002 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:17.003 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:17.003 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:17.004 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:01:17.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:01:17.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:17.005 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:01:17.112 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:17.111+0000 7f737cf478c0 -1 Falling back to public interface 2026-02-22T22:01:17.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:18.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:18.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:18.207 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:01:18.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:01:18.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:18.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:01:18.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:19.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:19.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:19.409 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:01:19.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:01:19.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:19.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:01:19.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:20.351 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:20.351+0000 7f737cf478c0 -1 osd.3 0 log_to_monitors true 2026-02-22T22:01:20.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:20.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:20.610 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:01:20.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:01:20.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:20.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:01:20.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:21.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:21.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:21.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:01:21.835 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:01:21.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:21.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:01:22.037 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/64051430,v1:127.0.0.1:6827/64051430] [v2:127.0.0.1:6828/64051430,v1:127.0.0.1:6829/64051430] exists,up 271adbe0-86e7-4264-8742-2c3415251445 2026-02-22T22:01:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:01:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:01:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:01:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:01:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-22T22:01:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:01:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:01:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2026-02-22T22:01:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:01:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-22T22:01:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2026-02-22T22:01:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:01:22.040 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:01:22.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:01:22.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:01:22.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:01:22.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:01:22.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:01:22.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:01:22.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:01:22.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:01:22.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:01:22.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:01:22.041 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:01:22.042 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:01:22.042 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-22T22:01:22.042 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:01:22.042 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6332da06-3ef6-4284-abe4-dc8db9113ba7 2026-02-22T22:01:22.042 INFO:tasks.workunit.client.0.trial034.stdout:add osd4 6332da06-3ef6-4284-abe4-dc8db9113ba7 2026-02-22T22:01:22.042 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 6332da06-3ef6-4284-abe4-dc8db9113ba7' 2026-02-22T22:01:22.042 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:01:22.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyfJtpO49fAxAAw6DXpXFFOrBQ5uf9AVqT3A== 2026-02-22T22:01:22.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyfJtpO49fAxAAw6DXpXFFOrBQ5uf9AVqT3A=="}' 2026-02-22T22:01:22.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6332da06-3ef6-4284-abe4-dc8db9113ba7 -i td/test-erasure-eio/4/new.json 2026-02-22T22:01:22.265 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:01:22.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-22T22:01:22.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyfJtpO49fAxAAw6DXpXFFOrBQ5uf9AVqT3A== --osd-uuid 6332da06-3ef6-4284-abe4-dc8db9113ba7 2026-02-22T22:01:22.294 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:22.295+0000 7fa9e88658c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:22.296 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:22.295+0000 7fa9e88658c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:22.297 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:22.295+0000 7fa9e88658c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:22.297 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:22.295+0000 7fa9e88658c0 -1 bdev(0x555ae3959c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-22T22:01:22.298 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:22.295+0000 7fa9e88658c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-22T22:01:24.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-22T22:01:24.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:01:24.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2026-02-22T22:01:24.593 INFO:tasks.workunit.client.0.trial034.stdout:adding osd4 key to auth repository 2026-02-22T22:01:24.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:01:24.870 INFO:tasks.workunit.client.0.trial034.stdout:start osd.4 2026-02-22T22:01:24.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2026-02-22T22:01:24.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:01:24.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:01:24.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:01:24.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:01:24.890 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:24.886+0000 7f4ac10d48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:24.891 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:24.890+0000 7f4ac10d48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:24.892 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:24.890+0000 7f4ac10d48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:25.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2026-02-22T22:01:25.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:01:25.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2026-02-22T22:01:25.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:01:25.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:01:25.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:25.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:01:25.069 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:01:25.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:25.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:01:25.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:26.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:26.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:26.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:01:26.271 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:01:26.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:26.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:01:26.392 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:26.390+0000 7f4ac10d48c0 -1 Falling back to public interface 2026-02-22T22:01:26.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:27.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:27.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:27.475 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:01:27.476 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:01:27.476 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:27.476 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:01:27.680 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:28.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:28.681 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:28.682 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:01:28.682 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:01:28.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:28.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:01:28.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:29.384 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:29.382+0000 7f4ac10d48c0 -1 osd.4 0 log_to_monitors true 2026-02-22T22:01:29.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:29.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:29.885 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:01:29.886 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:01:29.886 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:29.886 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:01:30.090 INFO:tasks.workunit.client.0.trial034.stdout:osd.4 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3599728388,v1:127.0.0.1:6835/3599728388] [v2:127.0.0.1:6836/3599728388,v1:127.0.0.1:6837/3599728388] exists,up 6332da06-3ef6-4284-abe4-dc8db9113ba7 2026-02-22T22:01:30.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:01:30.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:01:30.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:01:30.090 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T22:01:30.091 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:01:30.091 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:01:30.091 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:01:30.091 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:01:30.091 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:01:30.091 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T22:01:30.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T22:01:30.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T22:01:30.154 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:01:30.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T22:01:30.162 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:00:50.371+0000 7f1878c778c0 0 load: jerasure load: lrc 2026-02-22T22:01:30.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:01:30.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:621: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2026-02-22T22:01:30.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2026-02-22T22:01:30.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:01:30.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T22:01:30.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-22T22:01:30.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T22:01:30.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-22T22:01:30.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T22:01:30.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-22T22:01:30.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T22:01:30.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T22:01:30.744 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T22:01:30.754 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:01:31.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T22:01:31.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:01:31.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:01:31.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:01:31.757 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:01:31.757 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:01:31.757 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:01:31.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:01:31.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:01:31.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:01:31.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:01:31.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:01:31.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:01:31.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:01:31.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:01:31.863 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:01:32.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:01:32.065 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:01:32.065 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:01:32.065 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:01:32.065 INFO:tasks.workunit.client.0.trial034.stderr:4' 2026-02-22T22:01:32.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:01:32.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:01:32.065 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:01:32.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803786 2026-02-22T22:01:32.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803786 2026-02-22T22:01:32.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803786' 2026-02-22T22:01:32.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:01:32.141 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:01:32.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607560 2026-02-22T22:01:32.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607560 2026-02-22T22:01:32.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803786 1-51539607560' 2026-02-22T22:01:32.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:01:32.214 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:01:32.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=73014444039 2026-02-22T22:01:32.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 73014444039 2026-02-22T22:01:32.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803786 1-51539607560 2-73014444039' 2026-02-22T22:01:32.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:01:32.288 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:01:32.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182405 2026-02-22T22:01:32.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182405 2026-02-22T22:01:32.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803786 1-51539607560 2-73014444039 3-107374182405' 2026-02-22T22:01:32.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:01:32.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T22:01:32.439 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=141733920771 2026-02-22T22:01:32.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 141733920771 2026-02-22T22:01:32.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803786 1-51539607560 2-73014444039 3-107374182405 4-141733920771' 2026-02-22T22:01:32.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:01:32.441 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803786 2026-02-22T22:01:32.441 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:01:32.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:01:32.442 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803786 2026-02-22T22:01:32.442 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:01:32.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803786 2026-02-22T22:01:32.444 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803786 2026-02-22T22:01:32.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803786' 2026-02-22T22:01:32.444 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:01:32.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803784 -lt 25769803786 2026-02-22T22:01:32.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:01:33.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:01:33.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:01:33.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803786 -lt 25769803786 2026-02-22T22:01:33.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:01:33.836 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607560 2026-02-22T22:01:33.836 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:01:33.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:01:33.838 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607560 2026-02-22T22:01:33.838 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:01:33.839 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607560 2026-02-22T22:01:33.839 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607560 2026-02-22T22:01:33.840 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607560' 2026-02-22T22:01:33.840 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:01:34.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607561 -lt 51539607560 2026-02-22T22:01:34.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:01:34.038 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-73014444039 2026-02-22T22:01:34.039 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:01:34.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:01:34.041 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-73014444039 2026-02-22T22:01:34.041 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:01:34.042 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=73014444039 2026-02-22T22:01:34.042 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 73014444039 2026-02-22T22:01:34.043 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 73014444039' 2026-02-22T22:01:34.043 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:01:34.244 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 73014444039 -lt 73014444039 2026-02-22T22:01:34.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:01:34.245 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182405 2026-02-22T22:01:34.246 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:01:34.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:01:34.247 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182405 2026-02-22T22:01:34.247 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:01:34.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182405 2026-02-22T22:01:34.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182405' 2026-02-22T22:01:34.249 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 107374182405 2026-02-22T22:01:34.249 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:01:34.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182405 -lt 107374182405 2026-02-22T22:01:34.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:01:34.450 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-141733920771 2026-02-22T22:01:34.450 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:01:34.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T22:01:34.452 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-141733920771 2026-02-22T22:01:34.452 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:01:34.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=141733920771 2026-02-22T22:01:34.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 141733920771' 2026-02-22T22:01:34.454 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 141733920771 2026-02-22T22:01:34.454 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T22:01:34.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 141733920771 -lt 141733920771 2026-02-22T22:01:34.661 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:01:34.661 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:01:34.661 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:01:34.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:01:34.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:01:34.913 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:01:34.913 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:01:34.914 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:01:34.914 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:01:34.914 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:01:34.914 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:01:35.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:01:35.104 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:01:35.105 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:01:35.105 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:01:35.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:01:35.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:01:35.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:01:35.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_recovery_unfound: ceph pg dump pgs 2026-02-22T22:01:35.543 INFO:tasks.workunit.client.0.trial034.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T22:01:35.544 INFO:tasks.workunit.client.0.trial034.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:31.957872+0000 0'0 39:12 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-22T22:01:30.698678+0000 0'0 2026-02-22T22:01:30.698678+0000 0 0 periodic scrub scheduled @ 2026-02-24T09:24:31.674362+0000 0 0 2026-02-22T22:01:35.544 INFO:tasks.workunit.client.0.trial034.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:30.588693+0000 0'0 40:20 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-22T22:00:44.736776+0000 0'0 2026-02-22T22:00:44.736776+0000 0 0 periodic scrub scheduled @ 2026-02-23T22:48:10.090143+0000 0 0 2026-02-22T22:01:35.544 INFO:tasks.workunit.client.0.trial034.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:30.485065+0000 0'0 40:89 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-22T22:00:44.736776+0000 0'0 2026-02-22T22:00:44.736776+0000 0 0 periodic scrub scheduled @ 2026-02-24T06:38:06.551057+0000 0 0 2026-02-22T22:01:35.544 INFO:tasks.workunit.client.0.trial034.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:30.332968+0000 0'0 40:20 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-22T22:00:44.736776+0000 0'0 2026-02-22T22:00:44.736776+0000 0 0 periodic scrub scheduled @ 2026-02-24T04:49:59.063631+0000 0 0 2026-02-22T22:01:35.544 INFO:tasks.workunit.client.0.trial034.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:11.711416+0000 0'0 39:60 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-22T22:00:44.736776+0000 0'0 2026-02-22T22:00:44.736776+0000 0 0 periodic scrub scheduled @ 2026-02-24T06:20:51.096242+0000 0 0 2026-02-22T22:01:35.544 INFO:tasks.workunit.client.0.trial034.stdout: 2026-02-22T22:01:35.544 INFO:tasks.workunit.client.0.trial034.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T22:01:35.545 INFO:tasks.workunit.client.0.trial034.stderr:dumped pgs 2026-02-22T22:01:35.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:626: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-22T22:01:35.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:01:35.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:01:35.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-22T22:01:35.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:01:35.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:01:35.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:01:35.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:01:35.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:01:35.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:01:35.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:01:35.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:01:35.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:35.580 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2026-02-22T22:01:35.581 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:01:35.581 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2026-02-22T22:01:35.581 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-22T22:01:35.581 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:01:35.785 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:01:35.785 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:01:35.786 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:01:35.786 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:01:35.786 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:01:35.786 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2026-02-22T22:01:35.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: initial_osds=('3' '1' '4' '0' '2') 2026-02-22T22:01:35.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: local -a initial_osds 2026-02-22T22:01:35.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local last_osd=2 2026-02-22T22:01:35.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-22T22:01:35.786 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:01:35.787 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:01:35.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:01:35.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:01:35.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:01:37.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:01:37.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: ceph osd down 2 2026-02-22T22:01:37.293 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 is already down. 2026-02-22T22:01:37.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:632: TEST_ec_recovery_unfound: ceph osd out 2 2026-02-22T22:01:37.560 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 is already out. 2026-02-22T22:01:37.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:634: TEST_ec_recovery_unfound: ceph pg dump pgs 2026-02-22T22:01:37.773 INFO:tasks.workunit.client.0.trial034.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T22:01:37.774 INFO:tasks.workunit.client.0.trial034.stdout:2.0 1 0 1 0 0 4096 0 0 1 0 1 active+undersized+degraded 2026-02-22T22:01:35.913121+0000 40'1 42:25 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2026-02-22T22:01:30.698678+0000 0'0 2026-02-22T22:01:30.698678+0000 0 0 no scrub is scheduled 0 0 2026-02-22T22:01:37.774 INFO:tasks.workunit.client.0.trial034.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:30.588693+0000 0'0 40:20 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-22T22:00:44.736776+0000 0'0 2026-02-22T22:00:44.736776+0000 0 0 periodic scrub scheduled @ 2026-02-23T22:48:10.090143+0000 0 0 2026-02-22T22:01:37.774 INFO:tasks.workunit.client.0.trial034.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:30.485065+0000 0'0 40:89 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-22T22:00:44.736776+0000 0'0 2026-02-22T22:00:44.736776+0000 0 0 periodic scrub scheduled @ 2026-02-24T06:38:06.551057+0000 0 0 2026-02-22T22:01:37.774 INFO:tasks.workunit.client.0.trial034.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:30.332968+0000 0'0 40:20 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-22T22:00:44.736776+0000 0'0 2026-02-22T22:00:44.736776+0000 0 0 periodic scrub scheduled @ 2026-02-24T04:49:59.063631+0000 0 0 2026-02-22T22:01:37.774 INFO:tasks.workunit.client.0.trial034.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:11.711416+0000 0'0 39:60 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-22T22:00:44.736776+0000 0'0 2026-02-22T22:00:44.736776+0000 0 0 periodic scrub scheduled @ 2026-02-24T06:20:51.096242+0000 0 0 2026-02-22T22:01:37.774 INFO:tasks.workunit.client.0.trial034.stdout: 2026-02-22T22:01:37.774 INFO:tasks.workunit.client.0.trial034.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T22:01:37.774 INFO:tasks.workunit.client.0.trial034.stderr:dumped pgs 2026-02-22T22:01:37.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2026-02-22T22:01:37.783 INFO:tasks.workunit.client.0.trial034.stderr:4+0 records in 2026-02-22T22:01:37.784 INFO:tasks.workunit.client.0.trial034.stderr:4+0 records out 2026-02-22T22:01:37.784 INFO:tasks.workunit.client.0.trial034.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000111316 s, 36.8 MB/s 2026-02-22T22:01:37.784 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: seq 1 100 2026-02-22T22:01:37.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:37.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:37.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:37.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:37.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:37.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:37.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:37.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:37.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:37.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:37.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:37.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:37.928 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:37.928 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:37.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:37.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:37.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:37.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.057 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.057 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.083 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.083 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.447 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.447 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.764 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:38.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:38.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.043 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.043 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.805 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.841 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.841 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:39.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:39.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:40.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:40.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:40.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:40.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:40.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:40.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:40.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:40.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:40.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:40.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:40.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:01:40.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2026-02-22T22:01:40.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2026-02-22T22:01:40.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T22:01:40.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T22:01:40.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T22:01:40.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T22:01:40.160 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T22:01:40.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T22:01:40.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=obj75 2026-02-22T22:01:40.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T22:01:40.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T22:01:40.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T22:01:40.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=0 2026-02-22T22:01:40.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T22:01:40.161 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure obj75 2026-02-22T22:01:40.161 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:01:40.161 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2026-02-22T22:01:40.161 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2026-02-22T22:01:40.162 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:01:40.359 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:01:40.359 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:01:40.359 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:01:40.360 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:01:40.360 INFO:tasks.workunit.client.0.trial034.stderr:2147483647' 2026-02-22T22:01:40.360 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2026-02-22T22:01:40.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2026-02-22T22:01:40.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T22:01:40.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=3 2026-02-22T22:01:40.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T22:01:40.360 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/3/type 2026-02-22T22:01:40.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T22:01:40.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-22T22:01:40.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T22:01:40.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2026-02-22T22:01:40.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T22:01:40.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T22:01:40.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T22:01:40.363 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2026-02-22T22:01:40.363 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T22:01:40.363 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T22:01:40.363 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:01:40.363 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:01:40.363 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:01:40.364 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T22:01:40.364 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-22T22:01:40.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T22:01:40.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T22:01:40.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T22:01:40.424 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.3 2026-02-22T22:01:40.424 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T22:01:40.425 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T22:01:40.425 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:01:40.426 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:01:40.426 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:01:40.426 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T22:01:40.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T22:01:40.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2026-02-22T22:01:40.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:643: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2026-02-22T22:01:40.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T22:01:40.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T22:01:40.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T22:01:40.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T22:01:40.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T22:01:40.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T22:01:40.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=obj75 2026-02-22T22:01:40.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T22:01:40.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T22:01:40.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T22:01:40.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=1 2026-02-22T22:01:40.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T22:01:40.486 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure obj75 2026-02-22T22:01:40.487 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:01:40.487 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2026-02-22T22:01:40.487 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2026-02-22T22:01:40.487 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:01:40.687 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:01:40.687 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:01:40.687 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:01:40.687 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:01:40.688 INFO:tasks.workunit.client.0.trial034.stderr:2147483647' 2026-02-22T22:01:40.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2026-02-22T22:01:40.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2026-02-22T22:01:40.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T22:01:40.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=1 2026-02-22T22:01:40.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T22:01:40.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/1/type 2026-02-22T22:01:40.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T22:01:40.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-22T22:01:40.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T22:01:40.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2026-02-22T22:01:40.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T22:01:40.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T22:01:40.690 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T22:01:40.690 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2026-02-22T22:01:40.690 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2026-02-22T22:01:40.690 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2026-02-22T22:01:40.691 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:01:40.692 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:01:40.692 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:01:40.692 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.1.asok 2026-02-22T22:01:40.692 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-22T22:01:40.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T22:01:40.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T22:01:40.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T22:01:40.751 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.1 2026-02-22T22:01:40.751 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2026-02-22T22:01:40.751 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2026-02-22T22:01:40.751 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:01:40.751 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:01:40.751 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:01:40.752 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.1.asok 2026-02-22T22:01:40.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T22:01:40.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2026-02-22T22:01:40.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:645: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2026-02-22T22:01:40.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:01:40.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:01:40.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T22:01:40.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:01:40.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:01:40.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:01:40.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:01:40.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:01:40.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:01:40.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:01:40.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:01:40.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:01:40.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:01:40.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:01:40.811 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:01:40.811 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:01:40.811 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:01:40.811 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:01:40.811 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:01:40.811 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:01:40.811 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:01:40.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:01:40.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:01:40.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:01:40.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:01:40.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:01:40.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:01:40.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:01:40.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:01:40.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:01:40.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:01:40.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:01:40.813 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T22:01:40.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T22:01:40.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:01:40.814 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-22T22:01:40.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T22:01:40.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:01:40.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:01:40.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:01:40.829 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:40.826+0000 7f8bbe0d38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:40.830 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:40.830+0000 7f8bbe0d38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:40.831 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:40.830+0000 7f8bbe0d38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:01:41.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T22:01:41.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:01:41.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:01:41.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:01:41.005 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:01:41.005 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:41.005 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:01:41.005 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:01:41.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:41.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:01:41.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:42.064 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:42.062+0000 7f8bbe0d38c0 -1 Falling back to public interface 2026-02-22T22:01:42.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:42.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:42.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:01:42.212 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:01:42.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:42.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:01:42.407 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:43.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:43.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:43.408 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:01:43.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:01:43.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:43.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:01:43.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:44.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:44.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:44.610 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:01:44.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:01:44.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:44.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:01:44.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:01:45.348 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:01:45.346+0000 7f8bbe0d38c0 -1 osd.2 40 log_to_monitors true 2026-02-22T22:01:45.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:01:45.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:01:45.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:01:45.809 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:01:45.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:01:45.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:01:46.006 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up out weight 0 up_from 46 up_thru 17 down_at 41 last_clean_interval [17,40) [v2:127.0.0.1:6818/1640565112,v1:127.0.0.1:6819/1640565112] [v2:127.0.0.1:6820/1640565112,v1:127.0.0.1:6821/1640565112] exists,up bf4fa814-1924-4b4e-804a-17051ab5081e 2026-02-22T22:01:46.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:01:46.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:01:46.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:01:46.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: ceph osd in 2 2026-02-22T22:01:46.253 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 is already in. 2026-02-22T22:01:46.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: sleep 15 2026-02-22T22:02:01.265 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: seq 1 100 2026-02-22T22:02:01.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2026-02-22T22:02:01.267 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: get_state 2.0 2026-02-22T22:02:01.267 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2026-02-22T22:02:01.267 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2026-02-22T22:02:01.268 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2026-02-22T22:02:01.268 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2026-02-22T22:02:01.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2026-02-22T22:02:01.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2026-02-22T22:02:01.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: grep recovery_unfound 2026-02-22T22:02:01.463 INFO:tasks.workunit.client.0.trial034.stdout:active+recovery_unfound+degraded 2026-02-22T22:02:01.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2026-02-22T22:02:01.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: break 2026-02-22T22:02:01.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:660: TEST_ec_recovery_unfound: ceph pg dump pgs 2026-02-22T22:02:01.646 INFO:tasks.workunit.client.0.trial034.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T22:02:01.646 INFO:tasks.workunit.client.0.trial034.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2026-02-22T22:01:46.492612+0000 44'101 48:342 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2026-02-22T22:01:30.698678+0000 0'0 2026-02-22T22:01:30.698678+0000 0 0 no scrub is scheduled 0 0 2026-02-22T22:02:01.646 INFO:tasks.workunit.client.0.trial034.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:46.377091+0000 0'0 48:50 [4,1,2] 4 [4,1,2] 4 0'0 2026-02-22T22:00:44.736776+0000 0'0 2026-02-22T22:00:44.736776+0000 0 0 periodic scrub scheduled @ 2026-02-24T03:39:12.680868+0000 0 0 2026-02-22T22:02:01.646 INFO:tasks.workunit.client.0.trial034.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:30.485065+0000 0'0 48:104 [0,4,1] 0 [0,4,1] 0 0'0 2026-02-22T22:00:44.736776+0000 0'0 2026-02-22T22:00:44.736776+0000 0 0 periodic scrub scheduled @ 2026-02-24T06:38:06.551057+0000 0 0 2026-02-22T22:02:01.646 INFO:tasks.workunit.client.0.trial034.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:30.332968+0000 0'0 48:35 [4,3,0] 4 [4,3,0] 4 0'0 2026-02-22T22:00:44.736776+0000 0'0 2026-02-22T22:00:44.736776+0000 0 0 periodic scrub scheduled @ 2026-02-24T04:49:59.063631+0000 0 0 2026-02-22T22:02:01.646 INFO:tasks.workunit.client.0.trial034.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T22:01:46.379062+0000 0'0 48:93 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-22T22:00:44.736776+0000 0'0 2026-02-22T22:00:44.736776+0000 0 0 periodic scrub scheduled @ 2026-02-24T05:38:25.572229+0000 0 0 2026-02-22T22:02:01.646 INFO:tasks.workunit.client.0.trial034.stdout: 2026-02-22T22:02:01.646 INFO:tasks.workunit.client.0.trial034.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T22:02:01.647 INFO:tasks.workunit.client.0.trial034.stderr:dumped pgs 2026-02-22T22:02:01.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:661: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2026-02-22T22:02:01.738 INFO:tasks.workunit.client.0.trial034.stdout:{ 2026-02-22T22:02:01.739 INFO:tasks.workunit.client.0.trial034.stdout: "num_missing": 1, 2026-02-22T22:02:01.739 INFO:tasks.workunit.client.0.trial034.stdout: "num_unfound": 1, 2026-02-22T22:02:01.739 INFO:tasks.workunit.client.0.trial034.stdout: "objects": [ 2026-02-22T22:02:01.739 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.739 INFO:tasks.workunit.client.0.trial034.stdout: "oid": { 2026-02-22T22:02:01.739 INFO:tasks.workunit.client.0.trial034.stdout: "oid": "obj75", 2026-02-22T22:02:01.739 INFO:tasks.workunit.client.0.trial034.stdout: "key": "", 2026-02-22T22:02:01.739 INFO:tasks.workunit.client.0.trial034.stdout: "snapid": -2, 2026-02-22T22:02:01.739 INFO:tasks.workunit.client.0.trial034.stdout: "hash": 3565005477, 2026-02-22T22:02:01.739 INFO:tasks.workunit.client.0.trial034.stdout: "max": 0, 2026-02-22T22:02:01.740 INFO:tasks.workunit.client.0.trial034.stdout: "pool": 2, 2026-02-22T22:02:01.740 INFO:tasks.workunit.client.0.trial034.stdout: "namespace": "" 2026-02-22T22:02:01.740 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.740 INFO:tasks.workunit.client.0.trial034.stdout: "need": "44'76", 2026-02-22T22:02:01.740 INFO:tasks.workunit.client.0.trial034.stdout: "have": "0'0", 2026-02-22T22:02:01.740 INFO:tasks.workunit.client.0.trial034.stdout: "flags": "none", 2026-02-22T22:02:01.740 INFO:tasks.workunit.client.0.trial034.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2026-02-22T22:02:01.740 INFO:tasks.workunit.client.0.trial034.stdout: "locations": [ 2026-02-22T22:02:01.740 INFO:tasks.workunit.client.0.trial034.stdout: "0(3)", 2026-02-22T22:02:01.740 INFO:tasks.workunit.client.0.trial034.stdout: "4(2)" 2026-02-22T22:02:01.740 INFO:tasks.workunit.client.0.trial034.stdout: ] 2026-02-22T22:02:01.740 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.741 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.741 INFO:tasks.workunit.client.0.trial034.stdout: "state": "NotRecovering", 2026-02-22T22:02:01.741 INFO:tasks.workunit.client.0.trial034.stdout: "available_might_have_unfound": true, 2026-02-22T22:02:01.741 INFO:tasks.workunit.client.0.trial034.stdout: "might_have_unfound": [], 2026-02-22T22:02:01.741 INFO:tasks.workunit.client.0.trial034.stdout: "more": false 2026-02-22T22:02:01.741 INFO:tasks.workunit.client.0.trial034.stdout:} 2026-02-22T22:02:01.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: ceph pg 2.0 query 2026-02-22T22:02:01.823 INFO:tasks.workunit.client.0.trial034.stdout:{ 2026-02-22T22:02:01.823 INFO:tasks.workunit.client.0.trial034.stdout: "snap_trimq": "[]", 2026-02-22T22:02:01.823 INFO:tasks.workunit.client.0.trial034.stdout: "snap_trimq_len": 0, 2026-02-22T22:02:01.824 INFO:tasks.workunit.client.0.trial034.stdout: "state": "active+recovery_unfound+degraded", 2026-02-22T22:02:01.824 INFO:tasks.workunit.client.0.trial034.stdout: "epoch": 48, 2026-02-22T22:02:01.824 INFO:tasks.workunit.client.0.trial034.stdout: "up": [ 2026-02-22T22:02:01.824 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T22:02:01.824 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T22:02:01.824 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T22:02:01.824 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T22:02:01.824 INFO:tasks.workunit.client.0.trial034.stdout: 2 2026-02-22T22:02:01.824 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.824 INFO:tasks.workunit.client.0.trial034.stdout: "acting": [ 2026-02-22T22:02:01.824 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T22:02:01.825 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T22:02:01.825 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T22:02:01.825 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T22:02:01.825 INFO:tasks.workunit.client.0.trial034.stdout: 2 2026-02-22T22:02:01.825 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.825 INFO:tasks.workunit.client.0.trial034.stdout: "acting_recovery_backfill": [ 2026-02-22T22:02:01.825 INFO:tasks.workunit.client.0.trial034.stdout: "0(3)", 2026-02-22T22:02:01.825 INFO:tasks.workunit.client.0.trial034.stdout: "1(1)", 2026-02-22T22:02:01.825 INFO:tasks.workunit.client.0.trial034.stdout: "2(4)", 2026-02-22T22:02:01.826 INFO:tasks.workunit.client.0.trial034.stdout: "3(0)", 2026-02-22T22:02:01.826 INFO:tasks.workunit.client.0.trial034.stdout: "4(2)" 2026-02-22T22:02:01.826 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.826 INFO:tasks.workunit.client.0.trial034.stdout: "info": { 2026-02-22T22:02:01.826 INFO:tasks.workunit.client.0.trial034.stdout: "pgid": "2.0s0", 2026-02-22T22:02:01.826 INFO:tasks.workunit.client.0.trial034.stdout: "shared": "0", 2026-02-22T22:02:01.826 INFO:tasks.workunit.client.0.trial034.stdout: "last_update": "44'101", 2026-02-22T22:02:01.826 INFO:tasks.workunit.client.0.trial034.stdout: "last_complete": "44'75", 2026-02-22T22:02:01.826 INFO:tasks.workunit.client.0.trial034.stdout: "log_tail": "0'0", 2026-02-22T22:02:01.826 INFO:tasks.workunit.client.0.trial034.stdout: "last_user_version": 101, 2026-02-22T22:02:01.826 INFO:tasks.workunit.client.0.trial034.stdout: "last_backfill": "MAX", 2026-02-22T22:02:01.827 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [], 2026-02-22T22:02:01.827 INFO:tasks.workunit.client.0.trial034.stdout: "history": { 2026-02-22T22:02:01.827 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_created": 38, 2026-02-22T22:02:01.827 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_pool_created": 38, 2026-02-22T22:02:01.827 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 48, 2026-02-22T22:02:01.827 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_started": 47, 2026-02-22T22:02:01.827 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 39, 2026-02-22T22:02:01.827 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_clean": 38, 2026-02-22T22:02:01.827 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_split": 0, 2026-02-22T22:02:01.827 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_marked_full": 0, 2026-02-22T22:02:01.827 INFO:tasks.workunit.client.0.trial034.stdout: "same_up_since": 47, 2026-02-22T22:02:01.827 INFO:tasks.workunit.client.0.trial034.stdout: "same_interval_since": 47, 2026-02-22T22:02:01.828 INFO:tasks.workunit.client.0.trial034.stdout: "same_primary_since": 38, 2026-02-22T22:02:01.828 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T22:02:01.828 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.828 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T22:02:01.828 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.828 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.828 INFO:tasks.workunit.client.0.trial034.stdout: "prior_readable_until_ub": 0 2026-02-22T22:02:01.828 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.828 INFO:tasks.workunit.client.0.trial034.stdout: "stats": { 2026-02-22T22:02:01.828 INFO:tasks.workunit.client.0.trial034.stdout: "version": "44'101", 2026-02-22T22:02:01.828 INFO:tasks.workunit.client.0.trial034.stdout: "reported_seq": 343, 2026-02-22T22:02:01.828 INFO:tasks.workunit.client.0.trial034.stdout: "reported_epoch": 48, 2026-02-22T22:02:01.829 INFO:tasks.workunit.client.0.trial034.stdout: "state": "active+recovery_unfound+degraded", 2026-02-22T22:02:01.829 INFO:tasks.workunit.client.0.trial034.stdout: "last_fresh": "2026-02-22T22:01:46.492612+0000", 2026-02-22T22:02:01.829 INFO:tasks.workunit.client.0.trial034.stdout: "last_change": "2026-02-22T22:01:46.492612+0000", 2026-02-22T22:02:01.829 INFO:tasks.workunit.client.0.trial034.stdout: "last_active": "2026-02-22T22:01:46.492612+0000", 2026-02-22T22:02:01.829 INFO:tasks.workunit.client.0.trial034.stdout: "last_peered": "2026-02-22T22:01:46.492612+0000", 2026-02-22T22:02:01.829 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean": "2026-02-22T22:01:35.579219+0000", 2026-02-22T22:02:01.829 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_active": "2026-02-22T22:01:46.379614+0000", 2026-02-22T22:02:01.829 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_peered": "2026-02-22T22:01:46.379614+0000", 2026-02-22T22:02:01.829 INFO:tasks.workunit.client.0.trial034.stdout: "last_unstale": "2026-02-22T22:01:46.492612+0000", 2026-02-22T22:02:01.829 INFO:tasks.workunit.client.0.trial034.stdout: "last_undegraded": "2026-02-22T22:01:46.377171+0000", 2026-02-22T22:02:01.829 INFO:tasks.workunit.client.0.trial034.stdout: "last_fullsized": "2026-02-22T22:01:46.492612+0000", 2026-02-22T22:02:01.830 INFO:tasks.workunit.client.0.trial034.stdout: "mapping_epoch": 47, 2026-02-22T22:02:01.830 INFO:tasks.workunit.client.0.trial034.stdout: "log_start": "0'0", 2026-02-22T22:02:01.830 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_start": "0'0", 2026-02-22T22:02:01.830 INFO:tasks.workunit.client.0.trial034.stdout: "created": 38, 2026-02-22T22:02:01.830 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 39, 2026-02-22T22:02:01.830 INFO:tasks.workunit.client.0.trial034.stdout: "parent": "0.0", 2026-02-22T22:02:01.830 INFO:tasks.workunit.client.0.trial034.stdout: "parent_split_bits": 0, 2026-02-22T22:02:01.830 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T22:02:01.830 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.830 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T22:02:01.830 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.830 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "objects_scrubbed": 0, 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "log_size": 101, 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "log_dups_size": 0, 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_size": 101, 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "stats_invalid": false, 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "dirty_stats_invalid": false, 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "omap_stats_invalid": false, 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_stats_invalid": false, 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_bytes_stats_invalid": false, 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "pin_stats_invalid": false, 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "manifest_stats_invalid": false, 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrimq_len": 0, 2026-02-22T22:02:01.831 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_duration": 0, 2026-02-22T22:02:01.832 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_schedule": "no scrub is scheduled", 2026-02-22T22:02:01.832 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_duration": 0, 2026-02-22T22:02:01.832 INFO:tasks.workunit.client.0.trial034.stdout: "objects_trimmed": 0, 2026-02-22T22:02:01.832 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrim_duration": 0, 2026-02-22T22:02:01.832 INFO:tasks.workunit.client.0.trial034.stdout: "stat_sum": { 2026-02-22T22:02:01.832 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes": 413696, 2026-02-22T22:02:01.832 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects": 101, 2026-02-22T22:02:01.832 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_clones": 0, 2026-02-22T22:02:01.832 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_copies": 505, 2026-02-22T22:02:01.832 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing_on_primary": 1, 2026-02-22T22:02:01.832 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing": 0, 2026-02-22T22:02:01.832 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_degraded": 3, 2026-02-22T22:02:01.833 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_misplaced": 0, 2026-02-22T22:02:01.833 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_unfound": 1, 2026-02-22T22:02:01.833 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_dirty": 101, 2026-02-22T22:02:01.833 INFO:tasks.workunit.client.0.trial034.stdout: "num_whiteouts": 0, 2026-02-22T22:02:01.833 INFO:tasks.workunit.client.0.trial034.stdout: "num_read": 0, 2026-02-22T22:02:01.833 INFO:tasks.workunit.client.0.trial034.stdout: "num_read_kb": 0, 2026-02-22T22:02:01.833 INFO:tasks.workunit.client.0.trial034.stdout: "num_write": 101, 2026-02-22T22:02:01.833 INFO:tasks.workunit.client.0.trial034.stdout: "num_write_kb": 404, 2026-02-22T22:02:01.833 INFO:tasks.workunit.client.0.trial034.stdout: "num_scrub_errors": 0, 2026-02-22T22:02:01.833 INFO:tasks.workunit.client.0.trial034.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T22:02:01.833 INFO:tasks.workunit.client.0.trial034.stdout: "num_deep_scrub_errors": 0, 2026-02-22T22:02:01.833 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_recovered": 99, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_recovered": 405504, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_keys_recovered": 0, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_omap": 0, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush": 0, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_kb": 0, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict": 0, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_kb": 0, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_promote": 0, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_high": 0, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_low": 0, 2026-02-22T22:02:01.834 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_some": 0, 2026-02-22T22:02:01.835 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_full": 0, 2026-02-22T22:02:01.835 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_pinned": 0, 2026-02-22T22:02:01.835 INFO:tasks.workunit.client.0.trial034.stdout: "num_legacy_snapsets": 0, 2026-02-22T22:02:01.835 INFO:tasks.workunit.client.0.trial034.stdout: "num_large_omap_objects": 0, 2026-02-22T22:02:01.835 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_manifest": 0, 2026-02-22T22:02:01.835 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_bytes": 0, 2026-02-22T22:02:01.835 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_keys": 0, 2026-02-22T22:02:01.835 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_repaired": 0 2026-02-22T22:02:01.835 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.835 INFO:tasks.workunit.client.0.trial034.stdout: "up": [ 2026-02-22T22:02:01.835 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T22:02:01.835 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T22:02:01.836 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T22:02:01.836 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T22:02:01.836 INFO:tasks.workunit.client.0.trial034.stdout: 2 2026-02-22T22:02:01.836 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.836 INFO:tasks.workunit.client.0.trial034.stdout: "acting": [ 2026-02-22T22:02:01.836 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T22:02:01.836 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T22:02:01.836 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T22:02:01.836 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T22:02:01.836 INFO:tasks.workunit.client.0.trial034.stdout: 2 2026-02-22T22:02:01.836 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.836 INFO:tasks.workunit.client.0.trial034.stdout: "avail_no_missing": [ 2026-02-22T22:02:01.837 INFO:tasks.workunit.client.0.trial034.stdout: "0(3)", 2026-02-22T22:02:01.837 INFO:tasks.workunit.client.0.trial034.stdout: "4(2)" 2026-02-22T22:02:01.837 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.837 INFO:tasks.workunit.client.0.trial034.stdout: "object_location_counts": [ 2026-02-22T22:02:01.837 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.837 INFO:tasks.workunit.client.0.trial034.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2026-02-22T22:02:01.837 INFO:tasks.workunit.client.0.trial034.stdout: "objects": 100 2026-02-22T22:02:01.837 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.837 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.837 INFO:tasks.workunit.client.0.trial034.stdout: "shards": "0(3),4(2)", 2026-02-22T22:02:01.837 INFO:tasks.workunit.client.0.trial034.stdout: "objects": 1 2026-02-22T22:02:01.838 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.838 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.838 INFO:tasks.workunit.client.0.trial034.stdout: "blocked_by": [], 2026-02-22T22:02:01.838 INFO:tasks.workunit.client.0.trial034.stdout: "up_primary": 3, 2026-02-22T22:02:01.838 INFO:tasks.workunit.client.0.trial034.stdout: "acting_primary": 3, 2026-02-22T22:02:01.838 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [] 2026-02-22T22:02:01.838 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.838 INFO:tasks.workunit.client.0.trial034.stdout: "empty": 0, 2026-02-22T22:02:01.838 INFO:tasks.workunit.client.0.trial034.stdout: "dne": 0, 2026-02-22T22:02:01.838 INFO:tasks.workunit.client.0.trial034.stdout: "incomplete": 0, 2026-02-22T22:02:01.838 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 48, 2026-02-22T22:02:01.839 INFO:tasks.workunit.client.0.trial034.stdout: "hit_set_history": { 2026-02-22T22:02:01.839 INFO:tasks.workunit.client.0.trial034.stdout: "current_last_update": "0'0", 2026-02-22T22:02:01.839 INFO:tasks.workunit.client.0.trial034.stdout: "history": [] 2026-02-22T22:02:01.839 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.839 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.839 INFO:tasks.workunit.client.0.trial034.stdout: "peer_info": [ 2026-02-22T22:02:01.839 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.839 INFO:tasks.workunit.client.0.trial034.stdout: "peer": "0(3)", 2026-02-22T22:02:01.839 INFO:tasks.workunit.client.0.trial034.stdout: "pgid": "2.0s3", 2026-02-22T22:02:01.839 INFO:tasks.workunit.client.0.trial034.stdout: "shared": "3", 2026-02-22T22:02:01.840 INFO:tasks.workunit.client.0.trial034.stdout: "last_update": "44'101", 2026-02-22T22:02:01.840 INFO:tasks.workunit.client.0.trial034.stdout: "last_complete": "44'101", 2026-02-22T22:02:01.840 INFO:tasks.workunit.client.0.trial034.stdout: "log_tail": "0'0", 2026-02-22T22:02:01.840 INFO:tasks.workunit.client.0.trial034.stdout: "last_user_version": 101, 2026-02-22T22:02:01.840 INFO:tasks.workunit.client.0.trial034.stdout: "last_backfill": "MAX", 2026-02-22T22:02:01.840 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [], 2026-02-22T22:02:01.840 INFO:tasks.workunit.client.0.trial034.stdout: "history": { 2026-02-22T22:02:01.840 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_created": 38, 2026-02-22T22:02:01.840 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_pool_created": 38, 2026-02-22T22:02:01.840 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 48, 2026-02-22T22:02:01.841 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_started": 47, 2026-02-22T22:02:01.841 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 39, 2026-02-22T22:02:01.841 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_clean": 38, 2026-02-22T22:02:01.841 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_split": 0, 2026-02-22T22:02:01.841 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_marked_full": 0, 2026-02-22T22:02:01.841 INFO:tasks.workunit.client.0.trial034.stdout: "same_up_since": 47, 2026-02-22T22:02:01.841 INFO:tasks.workunit.client.0.trial034.stdout: "same_interval_since": 47, 2026-02-22T22:02:01.841 INFO:tasks.workunit.client.0.trial034.stdout: "same_primary_since": 38, 2026-02-22T22:02:01.841 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T22:02:01.841 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.842 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T22:02:01.842 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.842 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.842 INFO:tasks.workunit.client.0.trial034.stdout: "prior_readable_until_ub": 0 2026-02-22T22:02:01.842 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.842 INFO:tasks.workunit.client.0.trial034.stdout: "stats": { 2026-02-22T22:02:01.842 INFO:tasks.workunit.client.0.trial034.stdout: "version": "44'101", 2026-02-22T22:02:01.842 INFO:tasks.workunit.client.0.trial034.stdout: "reported_seq": 129, 2026-02-22T22:02:01.842 INFO:tasks.workunit.client.0.trial034.stdout: "reported_epoch": 44, 2026-02-22T22:02:01.842 INFO:tasks.workunit.client.0.trial034.stdout: "state": "active+undersized+degraded", 2026-02-22T22:02:01.843 INFO:tasks.workunit.client.0.trial034.stdout: "last_fresh": "2026-02-22T22:01:40.158597+0000", 2026-02-22T22:02:01.843 INFO:tasks.workunit.client.0.trial034.stdout: "last_change": "2026-02-22T22:01:35.913121+0000", 2026-02-22T22:02:01.843 INFO:tasks.workunit.client.0.trial034.stdout: "last_active": "2026-02-22T22:01:40.158597+0000", 2026-02-22T22:02:01.843 INFO:tasks.workunit.client.0.trial034.stdout: "last_peered": "2026-02-22T22:01:40.158597+0000", 2026-02-22T22:02:01.843 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean": "2026-02-22T22:01:35.579219+0000", 2026-02-22T22:02:01.843 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_active": "2026-02-22T22:01:35.913121+0000", 2026-02-22T22:02:01.844 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_peered": "2026-02-22T22:01:35.913121+0000", 2026-02-22T22:02:01.844 INFO:tasks.workunit.client.0.trial034.stdout: "last_unstale": "2026-02-22T22:01:40.158597+0000", 2026-02-22T22:02:01.844 INFO:tasks.workunit.client.0.trial034.stdout: "last_undegraded": "2026-02-22T22:01:35.911195+0000", 2026-02-22T22:02:01.844 INFO:tasks.workunit.client.0.trial034.stdout: "last_fullsized": "2026-02-22T22:01:35.911134+0000", 2026-02-22T22:02:01.844 INFO:tasks.workunit.client.0.trial034.stdout: "mapping_epoch": 47, 2026-02-22T22:02:01.844 INFO:tasks.workunit.client.0.trial034.stdout: "log_start": "0'0", 2026-02-22T22:02:01.845 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_start": "0'0", 2026-02-22T22:02:01.845 INFO:tasks.workunit.client.0.trial034.stdout: "created": 38, 2026-02-22T22:02:01.845 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 39, 2026-02-22T22:02:01.845 INFO:tasks.workunit.client.0.trial034.stdout: "parent": "0.0", 2026-02-22T22:02:01.845 INFO:tasks.workunit.client.0.trial034.stdout: "parent_split_bits": 0, 2026-02-22T22:02:01.845 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T22:02:01.845 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.845 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T22:02:01.845 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.845 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.845 INFO:tasks.workunit.client.0.trial034.stdout: "objects_scrubbed": 0, 2026-02-22T22:02:01.846 INFO:tasks.workunit.client.0.trial034.stdout: "log_size": 101, 2026-02-22T22:02:01.846 INFO:tasks.workunit.client.0.trial034.stdout: "log_dups_size": 0, 2026-02-22T22:02:01.846 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_size": 101, 2026-02-22T22:02:01.846 INFO:tasks.workunit.client.0.trial034.stdout: "stats_invalid": false, 2026-02-22T22:02:01.846 INFO:tasks.workunit.client.0.trial034.stdout: "dirty_stats_invalid": false, 2026-02-22T22:02:01.846 INFO:tasks.workunit.client.0.trial034.stdout: "omap_stats_invalid": false, 2026-02-22T22:02:01.846 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_stats_invalid": false, 2026-02-22T22:02:01.846 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_bytes_stats_invalid": false, 2026-02-22T22:02:01.846 INFO:tasks.workunit.client.0.trial034.stdout: "pin_stats_invalid": false, 2026-02-22T22:02:01.846 INFO:tasks.workunit.client.0.trial034.stdout: "manifest_stats_invalid": false, 2026-02-22T22:02:01.847 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrimq_len": 0, 2026-02-22T22:02:01.847 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_duration": 0, 2026-02-22T22:02:01.847 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-24T06:32:35.040555+0000", 2026-02-22T22:02:01.847 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_duration": 0, 2026-02-22T22:02:01.847 INFO:tasks.workunit.client.0.trial034.stdout: "objects_trimmed": 0, 2026-02-22T22:02:01.847 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrim_duration": 0, 2026-02-22T22:02:01.847 INFO:tasks.workunit.client.0.trial034.stdout: "stat_sum": { 2026-02-22T22:02:01.847 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes": 413696, 2026-02-22T22:02:01.847 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects": 101, 2026-02-22T22:02:01.847 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_clones": 0, 2026-02-22T22:02:01.847 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_copies": 505, 2026-02-22T22:02:01.847 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T22:02:01.848 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing": 0, 2026-02-22T22:02:01.848 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_degraded": 101, 2026-02-22T22:02:01.848 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_misplaced": 0, 2026-02-22T22:02:01.848 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_unfound": 0, 2026-02-22T22:02:01.848 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_dirty": 101, 2026-02-22T22:02:01.848 INFO:tasks.workunit.client.0.trial034.stdout: "num_whiteouts": 0, 2026-02-22T22:02:01.848 INFO:tasks.workunit.client.0.trial034.stdout: "num_read": 0, 2026-02-22T22:02:01.848 INFO:tasks.workunit.client.0.trial034.stdout: "num_read_kb": 0, 2026-02-22T22:02:01.848 INFO:tasks.workunit.client.0.trial034.stdout: "num_write": 101, 2026-02-22T22:02:01.848 INFO:tasks.workunit.client.0.trial034.stdout: "num_write_kb": 404, 2026-02-22T22:02:01.848 INFO:tasks.workunit.client.0.trial034.stdout: "num_scrub_errors": 0, 2026-02-22T22:02:01.848 INFO:tasks.workunit.client.0.trial034.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T22:02:01.849 INFO:tasks.workunit.client.0.trial034.stdout: "num_deep_scrub_errors": 0, 2026-02-22T22:02:01.849 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_recovered": 0, 2026-02-22T22:02:01.849 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_recovered": 0, 2026-02-22T22:02:01.849 INFO:tasks.workunit.client.0.trial034.stdout: "num_keys_recovered": 0, 2026-02-22T22:02:01.849 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_omap": 0, 2026-02-22T22:02:01.849 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T22:02:01.849 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T22:02:01.849 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush": 0, 2026-02-22T22:02:01.850 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_kb": 0, 2026-02-22T22:02:01.850 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict": 0, 2026-02-22T22:02:01.850 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_kb": 0, 2026-02-22T22:02:01.850 INFO:tasks.workunit.client.0.trial034.stdout: "num_promote": 0, 2026-02-22T22:02:01.850 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_high": 0, 2026-02-22T22:02:01.850 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_low": 0, 2026-02-22T22:02:01.851 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_some": 0, 2026-02-22T22:02:01.851 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_full": 0, 2026-02-22T22:02:01.851 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_pinned": 0, 2026-02-22T22:02:01.851 INFO:tasks.workunit.client.0.trial034.stdout: "num_legacy_snapsets": 0, 2026-02-22T22:02:01.851 INFO:tasks.workunit.client.0.trial034.stdout: "num_large_omap_objects": 0, 2026-02-22T22:02:01.851 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_manifest": 0, 2026-02-22T22:02:01.851 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_bytes": 0, 2026-02-22T22:02:01.851 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_keys": 0, 2026-02-22T22:02:01.851 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_repaired": 0 2026-02-22T22:02:01.851 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.851 INFO:tasks.workunit.client.0.trial034.stdout: "up": [ 2026-02-22T22:02:01.852 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T22:02:01.852 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T22:02:01.852 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T22:02:01.852 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T22:02:01.852 INFO:tasks.workunit.client.0.trial034.stdout: 2 2026-02-22T22:02:01.852 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.852 INFO:tasks.workunit.client.0.trial034.stdout: "acting": [ 2026-02-22T22:02:01.852 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T22:02:01.852 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T22:02:01.852 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T22:02:01.852 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T22:02:01.852 INFO:tasks.workunit.client.0.trial034.stdout: 2 2026-02-22T22:02:01.853 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.853 INFO:tasks.workunit.client.0.trial034.stdout: "avail_no_missing": [ 2026-02-22T22:02:01.853 INFO:tasks.workunit.client.0.trial034.stdout: "3(0)", 2026-02-22T22:02:01.853 INFO:tasks.workunit.client.0.trial034.stdout: "0(3)", 2026-02-22T22:02:01.853 INFO:tasks.workunit.client.0.trial034.stdout: "1(1)", 2026-02-22T22:02:01.853 INFO:tasks.workunit.client.0.trial034.stdout: "4(2)" 2026-02-22T22:02:01.853 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.853 INFO:tasks.workunit.client.0.trial034.stdout: "object_location_counts": [ 2026-02-22T22:02:01.853 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.853 INFO:tasks.workunit.client.0.trial034.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: "objects": 101 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: "blocked_by": [], 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: "up_primary": 3, 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: "acting_primary": 3, 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [] 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: "empty": 0, 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: "dne": 0, 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: "incomplete": 0, 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 48, 2026-02-22T22:02:01.854 INFO:tasks.workunit.client.0.trial034.stdout: "hit_set_history": { 2026-02-22T22:02:01.855 INFO:tasks.workunit.client.0.trial034.stdout: "current_last_update": "0'0", 2026-02-22T22:02:01.855 INFO:tasks.workunit.client.0.trial034.stdout: "history": [] 2026-02-22T22:02:01.855 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.855 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.855 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.855 INFO:tasks.workunit.client.0.trial034.stdout: "peer": "1(1)", 2026-02-22T22:02:01.855 INFO:tasks.workunit.client.0.trial034.stdout: "pgid": "2.0s1", 2026-02-22T22:02:01.855 INFO:tasks.workunit.client.0.trial034.stdout: "shared": "1", 2026-02-22T22:02:01.855 INFO:tasks.workunit.client.0.trial034.stdout: "last_update": "44'101", 2026-02-22T22:02:01.855 INFO:tasks.workunit.client.0.trial034.stdout: "last_complete": "44'101", 2026-02-22T22:02:01.855 INFO:tasks.workunit.client.0.trial034.stdout: "log_tail": "0'0", 2026-02-22T22:02:01.855 INFO:tasks.workunit.client.0.trial034.stdout: "last_user_version": 101, 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "last_backfill": "MAX", 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [], 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "history": { 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_created": 38, 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_pool_created": 38, 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 48, 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_started": 47, 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 39, 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_clean": 38, 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_split": 0, 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_marked_full": 0, 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "same_up_since": 47, 2026-02-22T22:02:01.856 INFO:tasks.workunit.client.0.trial034.stdout: "same_interval_since": 47, 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: "same_primary_since": 38, 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: "prior_readable_until_ub": 0 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: "stats": { 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: "version": "44'101", 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: "reported_seq": 129, 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: "reported_epoch": 44, 2026-02-22T22:02:01.857 INFO:tasks.workunit.client.0.trial034.stdout: "state": "active+undersized+degraded", 2026-02-22T22:02:01.858 INFO:tasks.workunit.client.0.trial034.stdout: "last_fresh": "2026-02-22T22:01:40.158597+0000", 2026-02-22T22:02:01.858 INFO:tasks.workunit.client.0.trial034.stdout: "last_change": "2026-02-22T22:01:35.913121+0000", 2026-02-22T22:02:01.858 INFO:tasks.workunit.client.0.trial034.stdout: "last_active": "2026-02-22T22:01:40.158597+0000", 2026-02-22T22:02:01.858 INFO:tasks.workunit.client.0.trial034.stdout: "last_peered": "2026-02-22T22:01:40.158597+0000", 2026-02-22T22:02:01.858 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean": "2026-02-22T22:01:35.579219+0000", 2026-02-22T22:02:01.858 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_active": "2026-02-22T22:01:35.913121+0000", 2026-02-22T22:02:01.858 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_peered": "2026-02-22T22:01:35.913121+0000", 2026-02-22T22:02:01.858 INFO:tasks.workunit.client.0.trial034.stdout: "last_unstale": "2026-02-22T22:01:40.158597+0000", 2026-02-22T22:02:01.858 INFO:tasks.workunit.client.0.trial034.stdout: "last_undegraded": "2026-02-22T22:01:35.911195+0000", 2026-02-22T22:02:01.858 INFO:tasks.workunit.client.0.trial034.stdout: "last_fullsized": "2026-02-22T22:01:35.911134+0000", 2026-02-22T22:02:01.858 INFO:tasks.workunit.client.0.trial034.stdout: "mapping_epoch": 47, 2026-02-22T22:02:01.858 INFO:tasks.workunit.client.0.trial034.stdout: "log_start": "0'0", 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_start": "0'0", 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "created": 38, 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 39, 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "parent": "0.0", 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "parent_split_bits": 0, 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "objects_scrubbed": 0, 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "log_size": 101, 2026-02-22T22:02:01.859 INFO:tasks.workunit.client.0.trial034.stdout: "log_dups_size": 0, 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_size": 101, 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "stats_invalid": false, 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "dirty_stats_invalid": false, 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "omap_stats_invalid": false, 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_stats_invalid": false, 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_bytes_stats_invalid": false, 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "pin_stats_invalid": false, 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "manifest_stats_invalid": false, 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrimq_len": 0, 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_duration": 0, 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-24T06:32:35.040555+0000", 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_duration": 0, 2026-02-22T22:02:01.860 INFO:tasks.workunit.client.0.trial034.stdout: "objects_trimmed": 0, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrim_duration": 0, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "stat_sum": { 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes": 413696, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects": 101, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_clones": 0, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_copies": 505, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing": 1, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_degraded": 101, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_misplaced": 0, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_unfound": 0, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_dirty": 101, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_whiteouts": 0, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_read": 0, 2026-02-22T22:02:01.861 INFO:tasks.workunit.client.0.trial034.stdout: "num_read_kb": 0, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_write": 101, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_write_kb": 404, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_scrub_errors": 0, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_deep_scrub_errors": 0, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_recovered": 0, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_recovered": 0, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_keys_recovered": 0, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_omap": 0, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush": 0, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_kb": 0, 2026-02-22T22:02:01.862 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_kb": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_promote": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_high": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_low": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_some": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_full": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_pinned": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_legacy_snapsets": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_large_omap_objects": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_manifest": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_bytes": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_keys": 0, 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_repaired": 0 2026-02-22T22:02:01.863 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: "up": [ 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: 2 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: "acting": [ 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: 2 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.864 INFO:tasks.workunit.client.0.trial034.stdout: "avail_no_missing": [ 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: "3(0)", 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: "0(3)", 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: "1(1)", 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: "4(2)" 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: "object_location_counts": [ 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: "objects": 101 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: "blocked_by": [], 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: "up_primary": 3, 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: "acting_primary": 3, 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [] 2026-02-22T22:02:01.865 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "empty": 0, 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "dne": 0, 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "incomplete": 0, 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 48, 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "hit_set_history": { 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "current_last_update": "0'0", 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "history": [] 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "peer": "2(4)", 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "pgid": "2.0s4", 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "shared": "4", 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "last_update": "44'101", 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "last_complete": "40'1", 2026-02-22T22:02:01.866 INFO:tasks.workunit.client.0.trial034.stdout: "log_tail": "0'0", 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "last_user_version": 1, 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "last_backfill": "MAX", 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [], 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "history": { 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_created": 38, 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_pool_created": 38, 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 48, 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_started": 47, 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 39, 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_clean": 38, 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_split": 0, 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_marked_full": 0, 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "same_up_since": 47, 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "same_interval_since": 47, 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "same_primary_since": 38, 2026-02-22T22:02:01.867 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "prior_readable_until_ub": 0 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "stats": { 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "version": "40'1", 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "reported_seq": 17, 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "reported_epoch": 40, 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "state": "active+clean", 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "last_fresh": "2026-02-22T22:01:35.579219+0000", 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "last_change": "2026-02-22T22:01:31.957872+0000", 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "last_active": "2026-02-22T22:01:35.579219+0000", 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "last_peered": "2026-02-22T22:01:35.579219+0000", 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean": "2026-02-22T22:01:35.579219+0000", 2026-02-22T22:02:01.868 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_active": "2026-02-22T22:01:31.957511+0000", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_peered": "2026-02-22T22:01:31.957511+0000", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "last_unstale": "2026-02-22T22:01:35.579219+0000", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "last_undegraded": "2026-02-22T22:01:35.579219+0000", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "last_fullsized": "2026-02-22T22:01:35.579219+0000", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "mapping_epoch": 47, 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "log_start": "0'0", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_start": "0'0", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "created": 38, 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 39, 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "parent": "0.0", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "parent_split_bits": 0, 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "objects_scrubbed": 0, 2026-02-22T22:02:01.869 INFO:tasks.workunit.client.0.trial034.stdout: "log_size": 1, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "log_dups_size": 0, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_size": 1, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "stats_invalid": false, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "dirty_stats_invalid": false, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "omap_stats_invalid": false, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_stats_invalid": false, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_bytes_stats_invalid": false, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "pin_stats_invalid": false, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "manifest_stats_invalid": false, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrimq_len": 0, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_duration": 0, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-23T23:12:47.831948+0000", 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_duration": 0, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "objects_trimmed": 0, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrim_duration": 0, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "stat_sum": { 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes": 4096, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects": 1, 2026-02-22T22:02:01.870 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_clones": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_copies": 5, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing": 1, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_degraded": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_misplaced": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_unfound": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_dirty": 1, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_whiteouts": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_read": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_read_kb": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_write": 1, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_write_kb": 4, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_scrub_errors": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_deep_scrub_errors": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_recovered": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_recovered": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_keys_recovered": 0, 2026-02-22T22:02:01.871 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_omap": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_kb": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_kb": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_promote": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_high": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_low": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_some": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_full": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_pinned": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_legacy_snapsets": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_large_omap_objects": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_manifest": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_bytes": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_keys": 0, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_repaired": 0 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.872 INFO:tasks.workunit.client.0.trial034.stdout: "up": [ 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: 2 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: "acting": [ 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: 2 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: "avail_no_missing": [], 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: "object_location_counts": [], 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: "blocked_by": [], 2026-02-22T22:02:01.873 INFO:tasks.workunit.client.0.trial034.stdout: "up_primary": 3, 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "acting_primary": 3, 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [] 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "empty": 0, 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "dne": 0, 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "incomplete": 0, 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 48, 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "hit_set_history": { 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "current_last_update": "0'0", 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "history": [] 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "peer": "4(2)", 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "pgid": "2.0s2", 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "shared": "2", 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "last_update": "44'101", 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "last_complete": "44'101", 2026-02-22T22:02:01.874 INFO:tasks.workunit.client.0.trial034.stdout: "log_tail": "0'0", 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_user_version": 101, 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_backfill": "MAX", 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [], 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "history": { 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_created": 38, 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "epoch_pool_created": 38, 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 48, 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_started": 47, 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 39, 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_interval_clean": 38, 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_split": 0, 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_marked_full": 0, 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "same_up_since": 47, 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "same_interval_since": 47, 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "same_primary_since": 38, 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.875 INFO:tasks.workunit.client.0.trial034.stdout: "prior_readable_until_ub": 0 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "stats": { 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "version": "44'101", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "reported_seq": 129, 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "reported_epoch": 44, 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "state": "active+undersized+degraded", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "last_fresh": "2026-02-22T22:01:40.158597+0000", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "last_change": "2026-02-22T22:01:35.913121+0000", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "last_active": "2026-02-22T22:01:40.158597+0000", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "last_peered": "2026-02-22T22:01:40.158597+0000", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean": "2026-02-22T22:01:35.579219+0000", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_active": "2026-02-22T22:01:35.913121+0000", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "last_became_peered": "2026-02-22T22:01:35.913121+0000", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "last_unstale": "2026-02-22T22:01:40.158597+0000", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "last_undegraded": "2026-02-22T22:01:35.911195+0000", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "last_fullsized": "2026-02-22T22:01:35.911134+0000", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "mapping_epoch": 47, 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "log_start": "0'0", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_start": "0'0", 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "created": 38, 2026-02-22T22:02:01.876 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_clean": 39, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "parent": "0.0", 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "parent_split_bits": 0, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub": "0'0", 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub": "0'0", 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "last_deep_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "last_clean_scrub_stamp": "2026-02-22T22:01:30.698678+0000", 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "objects_scrubbed": 0, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "log_size": 101, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "log_dups_size": 0, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "ondisk_log_size": 101, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "stats_invalid": false, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "dirty_stats_invalid": false, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "omap_stats_invalid": false, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_stats_invalid": false, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "hitset_bytes_stats_invalid": false, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "pin_stats_invalid": false, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "manifest_stats_invalid": false, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrimq_len": 0, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "last_scrub_duration": 0, 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_schedule": "periodic scrub scheduled @ 2026-02-24T06:32:35.040555+0000", 2026-02-22T22:02:01.877 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_duration": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "objects_trimmed": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "snaptrim_duration": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "stat_sum": { 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes": 413696, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects": 101, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_clones": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_object_copies": 505, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_missing": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_degraded": 101, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_misplaced": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_unfound": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_dirty": 101, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_whiteouts": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_read": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_read_kb": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_write": 101, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_write_kb": 404, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_scrub_errors": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_deep_scrub_errors": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_recovered": 0, 2026-02-22T22:02:01.878 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_recovered": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_keys_recovered": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_omap": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_kb": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_kb": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_promote": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_high": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_flush_mode_low": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_some": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_evict_mode_full": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_pinned": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_legacy_snapsets": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_large_omap_objects": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_manifest": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_bytes": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_omap_keys": 0, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "num_objects_repaired": 0 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.879 INFO:tasks.workunit.client.0.trial034.stdout: "up": [ 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: 2 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: "acting": [ 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: 3, 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: 1, 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: 4, 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: 0, 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: 2 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: "avail_no_missing": [ 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: "3(0)", 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: "0(3)", 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: "1(1)", 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: "4(2)" 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: "object_location_counts": [ 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: "shards": "0(3),1(1),3(0),4(2)", 2026-02-22T22:02:01.880 INFO:tasks.workunit.client.0.trial034.stdout: "objects": 101 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "blocked_by": [], 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "up_primary": 3, 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "acting_primary": 3, 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "purged_snaps": [] 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "empty": 0, 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "dne": 0, 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "incomplete": 0, 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "last_epoch_started": 48, 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "hit_set_history": { 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "current_last_update": "0'0", 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "history": [] 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "recovery_state": [ 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "name": "Started/Primary/Active", 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "enter_time": "2026-02-22T22:01:46.376575+0000", 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "might_have_unfound": [ 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.881 INFO:tasks.workunit.client.0.trial034.stdout: "osd": "0(3)", 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "status": "already probed" 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "osd": "1(1)", 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "status": "already probed" 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "osd": "2(4)", 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "status": "already probed" 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "osd": "4(2)", 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "status": "already probed" 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "recovery_progress": { 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "backfill_targets": [], 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "waiting_on_backfill": [], 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "last_backfill_started": "MIN", 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "backfill_info": { 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "begin": "MIN", 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "end": "MIN", 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "objects": [] 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.882 INFO:tasks.workunit.client.0.trial034.stdout: "peer_backfill_info": [], 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "backfills_in_flight": [], 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "recovering": [], 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "pg_backend": { 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "recovery_ops": [], 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "read_ops": [] 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: { 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "name": "Started", 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "enter_time": "2026-02-22T22:01:46.203231+0000" 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: } 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: ], 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "scrubber": { 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "active": false, 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "must_scrub": false, 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "must_deep_scrub": false, 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "must_repair": false, 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "need_auto": false, 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "scrub_reg_stamp": "2026-02-24T06:32:35.040555+0000", 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "schedule": "no scrub is scheduled" 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: }, 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout: "agent_state": {} 2026-02-22T22:02:01.883 INFO:tasks.workunit.client.0.trial034.stdout:} 2026-02-22T22:02:01.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2026-02-22T22:02:01.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: grep -q obj75 2026-02-22T22:02:01.915 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2026-02-22T22:02:01.915 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2026-02-22T22:02:01.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: check=true 2026-02-22T22:02:01.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:667: TEST_ec_recovery_unfound: test true == true 2026-02-22T22:02:01.994 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2026-02-22T22:02:01.994 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2026-02-22T22:02:02.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: check=0 2026-02-22T22:02:02.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: test 0 == 0 2026-02-22T22:02:02.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2026-02-22T22:02:07.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: test 124 = 124 2026-02-22T22:02:07.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:676: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2026-02-22T22:02:07.141 INFO:tasks.workunit.client.0.trial034.stderr:pg has no unfound objects 2026-02-22T22:02:07.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:678: TEST_ec_recovery_unfound: wait_for_clean 2026-02-22T22:02:07.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:02:07.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:02:07.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:02:07.149 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:02:07.150 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:02:07.150 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:02:07.151 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:02:07.151 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:02:07.151 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:02:07.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:02:07.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:02:07.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:02:07.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:02:07.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:02:07.261 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:02:07.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:02:07.462 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:02:07.462 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:02:07.462 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:02:07.462 INFO:tasks.workunit.client.0.trial034.stderr:4' 2026-02-22T22:02:07.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:02:07.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:02:07.462 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:02:07.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803795 2026-02-22T22:02:07.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803795 2026-02-22T22:02:07.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803795' 2026-02-22T22:02:07.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:02:07.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:02:07.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607569 2026-02-22T22:02:07.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607569 2026-02-22T22:02:07.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803795 1-51539607569' 2026-02-22T22:02:07.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:02:07.613 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:02:07.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=197568495623 2026-02-22T22:02:07.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 197568495623 2026-02-22T22:02:07.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803795 1-51539607569 2-197568495623' 2026-02-22T22:02:07.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:02:07.686 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:02:07.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182414 2026-02-22T22:02:07.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182414 2026-02-22T22:02:07.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803795 1-51539607569 2-197568495623 3-107374182414' 2026-02-22T22:02:07.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:02:07.761 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T22:02:07.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=141733920780 2026-02-22T22:02:07.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 141733920780 2026-02-22T22:02:07.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803795 1-51539607569 2-197568495623 3-107374182414 4-141733920780' 2026-02-22T22:02:07.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:02:07.837 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803795 2026-02-22T22:02:07.837 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:02:07.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:02:07.838 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803795 2026-02-22T22:02:07.838 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:02:07.840 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803795 2026-02-22T22:02:07.840 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803795 2026-02-22T22:02:07.840 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803795' 2026-02-22T22:02:07.841 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:02:08.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803793 -lt 25769803795 2026-02-22T22:02:08.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:02:09.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:02:09.050 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:02:09.244 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803795 -lt 25769803795 2026-02-22T22:02:09.244 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:02:09.245 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607569 2026-02-22T22:02:09.245 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:02:09.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:02:09.247 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607569 2026-02-22T22:02:09.247 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:02:09.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607569 2026-02-22T22:02:09.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607569' 2026-02-22T22:02:09.249 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607569 2026-02-22T22:02:09.249 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:02:09.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607570 -lt 51539607569 2026-02-22T22:02:09.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:02:09.462 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-197568495623 2026-02-22T22:02:09.462 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:02:09.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:02:09.464 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-197568495623 2026-02-22T22:02:09.464 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:02:09.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=197568495623 2026-02-22T22:02:09.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 197568495623' 2026-02-22T22:02:09.466 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 197568495623 2026-02-22T22:02:09.466 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:02:09.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 197568495623 -lt 197568495623 2026-02-22T22:02:09.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:02:09.671 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182414 2026-02-22T22:02:09.671 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:02:09.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:02:09.673 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182414 2026-02-22T22:02:09.673 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:02:09.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182414 2026-02-22T22:02:09.674 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 107374182414 2026-02-22T22:02:09.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182414' 2026-02-22T22:02:09.675 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:02:09.885 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182414 -lt 107374182414 2026-02-22T22:02:09.885 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:02:09.885 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-141733920780 2026-02-22T22:02:09.886 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:02:09.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T22:02:09.887 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-141733920780 2026-02-22T22:02:09.888 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:02:09.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=141733920780 2026-02-22T22:02:09.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 141733920780' 2026-02-22T22:02:09.889 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 141733920780 2026-02-22T22:02:09.890 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T22:02:10.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 141733920780 -lt 141733920780 2026-02-22T22:02:10.100 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:02:10.100 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:02:10.100 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:02:10.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:02:10.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:02:10.364 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:02:10.364 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:02:10.364 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:02:10.364 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:02:10.364 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:02:10.364 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:02:10.562 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:02:10.562 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:02:10.563 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:02:10.563 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:02:10.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:02:10.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:02:10.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:02:10.827 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: seq 1 100 2026-02-22T22:02:10.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:10.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2026-02-22T22:02:10.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2026-02-22T22:02:10.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:10.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:10.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2026-02-22T22:02:10.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2026-02-22T22:02:10.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:10.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:10.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2026-02-22T22:02:10.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2026-02-22T22:02:10.890 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:10.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:10.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2026-02-22T22:02:10.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2026-02-22T22:02:10.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:10.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:10.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2026-02-22T22:02:10.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2026-02-22T22:02:10.936 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:10.937 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:10.937 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2026-02-22T22:02:10.937 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2026-02-22T22:02:10.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:10.961 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:10.961 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2026-02-22T22:02:10.961 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2026-02-22T22:02:10.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:10.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:10.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2026-02-22T22:02:10.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2026-02-22T22:02:11.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2026-02-22T22:02:11.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2026-02-22T22:02:11.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2026-02-22T22:02:11.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2026-02-22T22:02:11.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.110 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.111 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.111 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2026-02-22T22:02:11.112 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2026-02-22T22:02:11.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.156 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2026-02-22T22:02:11.157 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.179 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2026-02-22T22:02:11.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2026-02-22T22:02:11.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2026-02-22T22:02:11.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2026-02-22T22:02:11.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2026-02-22T22:02:11.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2026-02-22T22:02:11.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2026-02-22T22:02:11.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2026-02-22T22:02:11.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.370 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.370 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2026-02-22T22:02:11.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2026-02-22T22:02:11.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.416 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.417 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.417 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2026-02-22T22:02:11.418 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2026-02-22T22:02:11.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2026-02-22T22:02:11.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2026-02-22T22:02:11.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2026-02-22T22:02:11.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.533 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.534 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.534 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2026-02-22T22:02:11.534 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2026-02-22T22:02:11.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2026-02-22T22:02:11.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.596 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2026-02-22T22:02:11.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2026-02-22T22:02:11.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2026-02-22T22:02:11.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.664 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2026-02-22T22:02:11.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2026-02-22T22:02:11.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2026-02-22T22:02:11.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.732 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2026-02-22T22:02:11.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.754 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.754 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2026-02-22T22:02:11.754 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.771 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.772 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2026-02-22T22:02:11.772 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2026-02-22T22:02:11.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2026-02-22T22:02:11.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2026-02-22T22:02:11.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.858 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.859 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.859 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2026-02-22T22:02:11.859 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2026-02-22T22:02:11.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2026-02-22T22:02:11.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2026-02-22T22:02:11.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2026-02-22T22:02:11.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2026-02-22T22:02:11.970 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2026-02-22T22:02:11.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2026-02-22T22:02:11.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2026-02-22T22:02:12.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2026-02-22T22:02:12.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2026-02-22T22:02:12.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2026-02-22T22:02:12.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.097 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2026-02-22T22:02:12.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2026-02-22T22:02:12.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2026-02-22T22:02:12.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2026-02-22T22:02:12.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.183 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.184 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.185 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2026-02-22T22:02:12.185 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2026-02-22T22:02:12.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2026-02-22T22:02:12.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2026-02-22T22:02:12.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2026-02-22T22:02:12.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2026-02-22T22:02:12.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2026-02-22T22:02:12.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2026-02-22T22:02:12.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.372 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2026-02-22T22:02:12.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2026-02-22T22:02:12.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2026-02-22T22:02:12.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.441 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2026-02-22T22:02:12.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2026-02-22T22:02:12.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2026-02-22T22:02:12.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.506 INFO:tasks.workunit.client.0.trial034.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2026-02-22T22:02:12.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2026-02-22T22:02:12.509 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.534 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.534 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2026-02-22T22:02:12.534 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.558 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.558 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2026-02-22T22:02:12.558 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2026-02-22T22:02:12.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2026-02-22T22:02:12.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2026-02-22T22:02:12.626 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.644 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.645 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.645 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2026-02-22T22:02:12.645 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2026-02-22T22:02:12.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2026-02-22T22:02:12.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2026-02-22T22:02:12.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.732 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2026-02-22T22:02:12.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.754 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.754 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2026-02-22T22:02:12.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2026-02-22T22:02:12.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2026-02-22T22:02:12.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2026-02-22T22:02:12.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2026-02-22T22:02:12.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2026-02-22T22:02:12.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2026-02-22T22:02:12.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2026-02-22T22:02:12.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2026-02-22T22:02:12.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2026-02-22T22:02:12.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2026-02-22T22:02:12.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:12.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:12.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2026-02-22T22:02:12.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2026-02-22T22:02:13.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:13.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:13.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2026-02-22T22:02:13.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2026-02-22T22:02:13.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:13.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:13.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2026-02-22T22:02:13.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2026-02-22T22:02:13.060 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:13.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2026-02-22T22:02:13.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2026-02-22T22:02:13.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2026-02-22T22:02:13.083 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:13.084 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:691: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2026-02-22T22:02:13.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2026-02-22T22:02:13.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:02:13.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T22:02:13.337 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T22:02:13.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T22:02:13.609 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T22:02:13.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T22:02:13.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:02:13.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:02:13.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:02:13.619 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:02:13.619 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:02:13.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:02:13.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:02:13.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:02:13.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:02:13.730 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:02:13.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:02:13.731 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:02:13.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:02:13.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:02:13.733 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:02:13.734 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:02:13.735 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:02:13.735 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:02:13.735 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:02:13.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:02:13.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:02:13.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:02:13.765 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:02:13.765 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:13.765 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:13.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:02:13.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:02:13.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:02:13.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T22:02:13.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T22:02:13.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T22:02:13.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T22:02:13.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:02:13.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:02:13.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:02:13.769 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:02:13.769 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:02:13.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:02:13.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:02:13.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:02:13.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:02:13.770 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:02:13.771 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:02:13.771 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:02:13.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:02:13.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:02:13.773 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:02:13.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:02:13.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:02:13.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:02:13.775 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:02:13.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:02:13.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:02:13.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:02:13.779 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:02:13.779 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:13.779 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:13.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:02:13.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:02:13.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:02:13.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T22:02:13.782 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:02:13.782 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:13.782 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:13.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T22:02:13.783 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:02:13.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:02:13.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:02:13.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T22:02:13.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T22:02:13.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T22:02:13.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:02:13.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:02:13.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:02:13.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T22:02:13.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T22:02:13.806 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:02:13.807 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:02:13.807 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:02:13.807 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:02:13.807 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:13.807 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:13.807 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:02:13.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:02:13.834 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:02:13.835 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:02:13.835 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:02:13.835 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:02:13.836 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:02:13.836 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:02:13.837 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:02:13.837 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:02:13.837 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:02:13.837 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:02:13.837 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:13.837 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:13.837 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:02:13.837 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:02:13.838 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T22:02:13.892 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:02:13.892 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:02:13.892 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:02:13.892 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:02:13.893 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:02:13.893 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:02:13.893 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:02:13.893 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:02:13.894 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:02:13.894 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:13.894 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:13.894 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:02:13.894 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:02:13.894 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T22:02:13.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T22:02:13.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T22:02:13.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:02:13.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:02:13.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:02:13.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T22:02:13.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:02:14.060 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:02:14.060 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:02:14.060 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:02:14.060 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:02:14.061 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:14.061 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:14.061 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:02:14.061 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:02:14.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:02:14.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T22:02:14.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:02:14.191 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T22:02:14.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:02:15.202 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T22:02:15.202 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:02:15.202 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:02:15.202 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:02:15.202 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:15.202 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:15.203 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:02:15.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T22:02:15.203 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T22:02:15.251 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:02:15.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T22:02:15.259 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:02:13.821+0000 7ffb274c1ec0 0 load: jerasure load: lrc 2026-02-22T22:02:15.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2026-02-22T22:02:15.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:390: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2026-02-22T22:02:15.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:391: TEST_ec_single_recovery_error: local objname=myobject 2026-02-22T22:02:15.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:393: TEST_ec_single_recovery_error: setup_osds 7 2026-02-22T22:02:15.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2026-02-22T22:02:15.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T22:02:15.262 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2026-02-22T22:02:15.262 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2026-02-22T22:02:15.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:02:15.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T22:02:15.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:02:15.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:02:15.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:02:15.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:02:15.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T22:02:15.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:02:15.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:02:15.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:02:15.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:02:15.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T22:02:15.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T22:02:15.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:02:15.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:02:15.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:02:15.271 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:02:15.271 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:02:15.271 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:02:15.271 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:02:15.272 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:15.272 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:15.272 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:02:15.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:02:15.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:02:15.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:02:15.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:02:15.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:02:15.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:02:15.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:02:15.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:02:15.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:02:15.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:02:15.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:02:15.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:02:15.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T22:02:15.274 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:02:15.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=29f80264-71a1-4f89-9bcd-ed7759935636 2026-02-22T22:02:15.275 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 29f80264-71a1-4f89-9bcd-ed7759935636 2026-02-22T22:02:15.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 29f80264-71a1-4f89-9bcd-ed7759935636' 2026-02-22T22:02:15.276 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:02:15.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDnfJtpIk+MEBAA1YfUl1iTDcwbYpYlAG3aMQ== 2026-02-22T22:02:15.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDnfJtpIk+MEBAA1YfUl1iTDcwbYpYlAG3aMQ=="}' 2026-02-22T22:02:15.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 29f80264-71a1-4f89-9bcd-ed7759935636 -i td/test-erasure-eio/0/new.json 2026-02-22T22:02:15.389 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:02:15.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T22:02:15.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnfJtpIk+MEBAA1YfUl1iTDcwbYpYlAG3aMQ== --osd-uuid 29f80264-71a1-4f89-9bcd-ed7759935636 2026-02-22T22:02:15.413 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:15.409+0000 7f4bb8ba28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:15.414 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:15.413+0000 7f4bb8ba28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:15.415 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:15.413+0000 7f4bb8ba28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:15.416 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:15.413+0000 7f4bb8ba28c0 -1 bdev(0x55f433699c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:02:15.416 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:15.413+0000 7f4bb8ba28c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T22:02:17.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T22:02:17.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:02:17.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:02:17.982 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T22:02:17.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:02:18.251 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T22:02:18.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:02:18.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:02:18.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:02:18.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:02:18.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:02:18.273 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:18.269+0000 7f2ab19658c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:18.275 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:18.273+0000 7f2ab19658c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:18.276 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:18.273+0000 7f2ab19658c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:18.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:02:18.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:02:18.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:02:18.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:02:18.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:02:18.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:18.451 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:02:18.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:02:18.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:18.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:02:18.647 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:19.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:19.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:19.649 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:02:19.649 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:02:19.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:19.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:02:19.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:20.068 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:20.065+0000 7f2ab19658c0 -1 Falling back to public interface 2026-02-22T22:02:20.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:20.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:20.845 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:02:20.845 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:02:20.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:20.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:02:21.038 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:22.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:02:22.039 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:02:22.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:22.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:02:22.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:23.126 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:23.121+0000 7f2ab19658c0 -1 osd.0 0 log_to_monitors true 2026-02-22T22:02:23.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:23.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:23.232 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:02:23.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:02:23.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:23.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:02:23.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:24.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:24.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:24.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:02:24.429 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:02:24.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:24.430 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:02:24.627 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2225922106,v1:127.0.0.1:6803/2225922106] [v2:127.0.0.1:6804/2225922106,v1:127.0.0.1:6805/2225922106] exists,up 29f80264-71a1-4f89-9bcd-ed7759935636 2026-02-22T22:02:24.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:02:24.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:02:24.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:02:24.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:02:24.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T22:02:24.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:02:24.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:02:24.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:02:24.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:02:24.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:02:24.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:02:24.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:02:24.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:02:24.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:02:24.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:02:24.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:02:24.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:02:24.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:02:24.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:02:24.632 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:02:24.632 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:02:24.632 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:02:24.632 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:02:24.633 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:24.633 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:24.633 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:02:24.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:02:24.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:02:24.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:02:24.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:02:24.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:02:24.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:02:24.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:02:24.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:02:24.633 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:02:24.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:02:24.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:02:24.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:02:24.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:02:24.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:02:24.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89995f1c-477d-436d-b2c2-d52d5a904e7a 2026-02-22T22:02:24.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 89995f1c-477d-436d-b2c2-d52d5a904e7a' 2026-02-22T22:02:24.636 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 89995f1c-477d-436d-b2c2-d52d5a904e7a 2026-02-22T22:02:24.636 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:02:24.644 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDwfJtpLYV4JhAA4KtfGV72zUttSYBE2wskvQ== 2026-02-22T22:02:24.645 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDwfJtpLYV4JhAA4KtfGV72zUttSYBE2wskvQ=="}' 2026-02-22T22:02:24.645 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89995f1c-477d-436d-b2c2-d52d5a904e7a -i td/test-erasure-eio/1/new.json 2026-02-22T22:02:24.852 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:02:24.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T22:02:24.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDwfJtpLYV4JhAA4KtfGV72zUttSYBE2wskvQ== --osd-uuid 89995f1c-477d-436d-b2c2-d52d5a904e7a 2026-02-22T22:02:24.887 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:24.885+0000 7ffbc197b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:24.889 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:24.885+0000 7ffbc197b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:24.890 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:24.885+0000 7ffbc197b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:24.891 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:24.889+0000 7ffbc197b8c0 -1 bdev(0x55dbdfe31c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:02:24.891 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:24.889+0000 7ffbc197b8c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T22:02:27.172 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T22:02:27.172 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:02:27.173 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:02:27.173 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T22:02:27.173 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:02:27.438 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T22:02:27.439 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:02:27.439 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:02:27.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:02:27.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:02:27.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:02:27.462 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:27.457+0000 7fa888d8b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:27.464 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:27.461+0000 7fa888d8b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:27.465 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:27.461+0000 7fa888d8b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:27.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:02:27.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:02:27.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:02:27.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:02:27.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:02:27.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:27.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:02:27.637 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:02:27.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:27.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:02:27.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:28.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:28.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:28.831 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:02:28.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:02:28.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:28.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:02:28.964 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:28.961+0000 7fa888d8b8c0 -1 Falling back to public interface 2026-02-22T22:02:29.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:30.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:30.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:30.025 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:02:30.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:02:30.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:30.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:02:30.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:31.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:31.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:31.217 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:02:31.218 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:02:31.218 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:31.218 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:02:31.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:32.200 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:32.197+0000 7fa888d8b8c0 -1 osd.1 0 log_to_monitors true 2026-02-22T22:02:32.415 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:32.415 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:32.415 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:02:32.416 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:02:32.416 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:32.416 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:02:32.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:33.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:33.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:33.635 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:02:33.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:02:33.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:33.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:02:33.826 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2888383901,v1:127.0.0.1:6811/2888383901] [v2:127.0.0.1:6812/2888383901,v1:127.0.0.1:6813/2888383901] exists,up 89995f1c-477d-436d-b2c2-d52d5a904e7a 2026-02-22T22:02:33.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:02:33.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:02:33.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:02:33.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:02:33.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:02:33.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:02:33.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:02:33.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:02:33.830 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:02:33.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=af0a4c2d-6b3a-4610-a450-2fb0c4a45a09 2026-02-22T22:02:33.831 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 af0a4c2d-6b3a-4610-a450-2fb0c4a45a09 2026-02-22T22:02:33.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 af0a4c2d-6b3a-4610-a450-2fb0c4a45a09' 2026-02-22T22:02:33.831 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:02:33.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5fJtpEVtbMhAAugIfCxq3FGTMnEV05JTluQ== 2026-02-22T22:02:33.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5fJtpEVtbMhAAugIfCxq3FGTMnEV05JTluQ=="}' 2026-02-22T22:02:33.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new af0a4c2d-6b3a-4610-a450-2fb0c4a45a09 -i td/test-erasure-eio/2/new.json 2026-02-22T22:02:34.044 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:02:34.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T22:02:34.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5fJtpEVtbMhAAugIfCxq3FGTMnEV05JTluQ== --osd-uuid af0a4c2d-6b3a-4610-a450-2fb0c4a45a09 2026-02-22T22:02:34.070 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:34.065+0000 7fa31224b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:34.071 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:34.069+0000 7fa31224b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:34.072 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:34.069+0000 7fa31224b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:34.073 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:34.069+0000 7fa31224b8c0 -1 bdev(0x55da08c09c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:02:34.073 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:34.069+0000 7fa31224b8c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T22:02:36.331 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T22:02:36.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:02:36.332 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T22:02:36.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:02:36.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:02:36.597 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T22:02:36.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:02:36.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:02:36.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:02:36.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:02:36.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:02:36.621 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:36.617+0000 7f2fb2a128c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:36.622 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:36.617+0000 7f2fb2a128c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:36.623 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:36.621+0000 7f2fb2a128c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:36.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:02:36.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:02:36.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:02:36.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:02:36.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:02:36.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:36.803 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:02:36.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:02:36.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:36.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:02:36.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:37.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:37.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:37.996 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:02:37.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:02:37.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:37.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:02:38.128 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:38.125+0000 7f2fb2a128c0 -1 Falling back to public interface 2026-02-22T22:02:38.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:39.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:39.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:39.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:02:39.201 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:02:39.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:39.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:02:39.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:40.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:40.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:40.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:02:40.398 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:02:40.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:40.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:02:40.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:41.139 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:41.137+0000 7f2fb2a128c0 -1 osd.2 0 log_to_monitors true 2026-02-22T22:02:41.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:41.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:41.592 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:02:41.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:02:41.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:41.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:02:41.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:42.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:42.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:42.793 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:02:42.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:02:42.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:42.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:02:42.990 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3248289601,v1:127.0.0.1:6819/3248289601] [v2:127.0.0.1:6820/3248289601,v1:127.0.0.1:6821/3248289601] exists,up af0a4c2d-6b3a-4610-a450-2fb0c4a45a09 2026-02-22T22:02:42.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:02:42.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:02:42.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:02:42.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:02:42.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T22:02:42.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:02:42.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:02:42.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T22:02:42.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:02:42.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:02:42.995 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:02:42.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:02:42.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:02:42.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:02:42.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:02:42.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:02:42.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:02:42.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:02:42.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:02:42.996 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:02:42.996 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:02:42.996 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:02:42.996 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:02:42.997 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:42.997 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:42.997 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:02:42.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:02:42.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:02:42.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:02:42.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:02:42.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:02:42.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:02:42.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:02:42.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:02:42.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:02:42.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:02:42.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:02:42.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:02:42.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:02:42.998 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:02:42.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5bc7ffa1-7192-46ab-8aee-3b7f09973b59 2026-02-22T22:02:42.998 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 5bc7ffa1-7192-46ab-8aee-3b7f09973b59 2026-02-22T22:02:42.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5bc7ffa1-7192-46ab-8aee-3b7f09973b59' 2026-02-22T22:02:42.999 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:02:43.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADfZtpzW6EABAADm5eZlxYkt04P4EalZ8Yuw== 2026-02-22T22:02:43.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADfZtpzW6EABAADm5eZlxYkt04P4EalZ8Yuw=="}' 2026-02-22T22:02:43.009 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5bc7ffa1-7192-46ab-8aee-3b7f09973b59 -i td/test-erasure-eio/3/new.json 2026-02-22T22:02:43.210 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:02:43.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T22:02:43.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADfZtpzW6EABAADm5eZlxYkt04P4EalZ8Yuw== --osd-uuid 5bc7ffa1-7192-46ab-8aee-3b7f09973b59 2026-02-22T22:02:43.235 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:43.233+0000 7f38700e08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:43.237 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:43.233+0000 7f38700e08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:43.238 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:43.233+0000 7f38700e08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:43.238 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:43.237+0000 7f38700e08c0 -1 bdev(0x55cfb9e2bc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T22:02:43.238 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:43.237+0000 7f38700e08c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T22:02:45.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T22:02:45.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:02:45.516 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T22:02:45.516 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T22:02:45.516 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:02:45.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T22:02:45.784 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T22:02:45.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:02:45.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:02:45.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:02:45.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:02:45.807 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:45.805+0000 7f2a8dcd98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:45.808 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:45.805+0000 7f2a8dcd98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:45.809 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:45.805+0000 7f2a8dcd98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:45.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T22:02:45.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:02:45.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:02:45.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:02:45.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:02:45.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:45.981 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:02:45.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:02:45.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:45.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:02:46.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:47.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:47.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:47.177 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:02:47.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:02:47.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:47.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:02:47.304 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:47.301+0000 7f2a8dcd98c0 -1 Falling back to public interface 2026-02-22T22:02:47.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:48.375 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:48.375 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:48.375 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:02:48.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:02:48.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:48.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:02:48.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:49.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:49.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:49.572 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:02:49.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:02:49.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:49.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:02:49.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:50.614 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:50.609+0000 7f2a8dcd98c0 -1 osd.3 0 log_to_monitors true 2026-02-22T22:02:50.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:50.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:50.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:02:50.769 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:02:50.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:50.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:02:50.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:51.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:51.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:51.981 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:02:51.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:02:51.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:51.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:02:52.175 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3780722122,v1:127.0.0.1:6827/3780722122] [v2:127.0.0.1:6828/3780722122,v1:127.0.0.1:6829/3780722122] exists,up 5bc7ffa1-7192-46ab-8aee-3b7f09973b59 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:02:52.176 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:02:52.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:02:52.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:02:52.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:02:52.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:02:52.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2026-02-22T22:02:52.179 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:02:52.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b97bd49e-53c2-4702-87ee-06ae2044af0a 2026-02-22T22:02:52.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 b97bd49e-53c2-4702-87ee-06ae2044af0a' 2026-02-22T22:02:52.180 INFO:tasks.workunit.client.0.trial034.stdout:add osd4 b97bd49e-53c2-4702-87ee-06ae2044af0a 2026-02-22T22:02:52.181 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:02:52.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAMfZtprx+WCxAAqicLE6ToIErhz2hbbcRcYA== 2026-02-22T22:02:52.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAMfZtprx+WCxAAqicLE6ToIErhz2hbbcRcYA=="}' 2026-02-22T22:02:52.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b97bd49e-53c2-4702-87ee-06ae2044af0a -i td/test-erasure-eio/4/new.json 2026-02-22T22:02:52.390 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:02:52.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2026-02-22T22:02:52.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMfZtprx+WCxAAqicLE6ToIErhz2hbbcRcYA== --osd-uuid b97bd49e-53c2-4702-87ee-06ae2044af0a 2026-02-22T22:02:52.415 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:52.412+0000 7fa2e84d38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:52.416 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:52.412+0000 7fa2e84d38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:52.417 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:52.412+0000 7fa2e84d38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:52.418 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:52.412+0000 7fa2e84d38c0 -1 bdev(0x5602bcd95c00 td/test-erasure-eio/4/block) open stat got: (1) Operation not permitted 2026-02-22T22:02:52.418 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:52.412+0000 7fa2e84d38c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2026-02-22T22:02:54.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2026-02-22T22:02:54.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:02:54.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2026-02-22T22:02:54.720 INFO:tasks.workunit.client.0.trial034.stdout:adding osd4 key to auth repository 2026-02-22T22:02:54.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:02:54.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2026-02-22T22:02:54.997 INFO:tasks.workunit.client.0.trial034.stdout:start osd.4 2026-02-22T22:02:54.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:02:54.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:02:54.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:02:54.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:02:55.018 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:55.016+0000 7f25a99ba8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:55.020 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:55.016+0000 7f25a99ba8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:55.021 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:55.016+0000 7f25a99ba8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:02:55.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2026-02-22T22:02:55.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:02:55.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2026-02-22T22:02:55.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:02:55.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:02:55.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:55.200 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:02:55.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:02:55.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:55.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:02:55.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:56.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:56.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:56.396 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:02:56.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:02:56.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:56.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:02:56.516 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:56.512+0000 7f25a99ba8c0 -1 Falling back to public interface 2026-02-22T22:02:56.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:57.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:57.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:57.593 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:02:57.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:02:57.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:57.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:02:57.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:58.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:58.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:58.789 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:02:58.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:02:58.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:58.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:02:58.987 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:02:59.521 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:02:59.516+0000 7f25a99ba8c0 -1 osd.4 0 log_to_monitors true 2026-02-22T22:02:59.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:02:59.989 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:02:59.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:02:59.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:02:59.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:02:59.990 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2026-02-22T22:03:00.190 INFO:tasks.workunit.client.0.trial034.stdout:osd.4 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3957991528,v1:127.0.0.1:6835/3957991528] [v2:127.0.0.1:6836/3957991528,v1:127.0.0.1:6837/3957991528] exists,up b97bd49e-53c2-4702-87ee-06ae2044af0a 2026-02-22T22:03:00.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:03:00.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:03:00.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:03:00.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:03:00.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:03:00.192 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:03:00.193 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:03:00.193 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:00.193 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:00.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:03:00.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:03:00.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:03:00.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:03:00.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:03:00.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:03:00.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:03:00.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:03:00.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:03:00.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:03:00.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:03:00.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:03:00.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:03:00.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2026-02-22T22:03:00.194 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:03:00.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7a9492b7-2aba-4520-952a-e5fbb7659aa4 2026-02-22T22:03:00.195 INFO:tasks.workunit.client.0.trial034.stdout:add osd5 7a9492b7-2aba-4520-952a-e5fbb7659aa4 2026-02-22T22:03:00.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 7a9492b7-2aba-4520-952a-e5fbb7659aa4' 2026-02-22T22:03:00.196 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:03:00.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUfZtpLoZwDBAAC0ZQYACddVfR/CbtTgQ13w== 2026-02-22T22:03:00.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUfZtpLoZwDBAAC0ZQYACddVfR/CbtTgQ13w=="}' 2026-02-22T22:03:00.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7a9492b7-2aba-4520-952a-e5fbb7659aa4 -i td/test-erasure-eio/5/new.json 2026-02-22T22:03:00.400 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:03:00.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2026-02-22T22:03:00.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUfZtpLoZwDBAAC0ZQYACddVfR/CbtTgQ13w== --osd-uuid 7a9492b7-2aba-4520-952a-e5fbb7659aa4 2026-02-22T22:03:00.426 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:00.420+0000 7f94ebbde8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:00.427 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:00.424+0000 7f94ebbde8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:00.428 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:00.424+0000 7f94ebbde8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:00.428 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:00.424+0000 7f94ebbde8c0 -1 bdev(0x55c7b0ad5c00 td/test-erasure-eio/5/block) open stat got: (1) Operation not permitted 2026-02-22T22:03:00.429 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:00.424+0000 7f94ebbde8c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2026-02-22T22:03:02.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2026-02-22T22:03:02.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:03:02.704 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2026-02-22T22:03:02.705 INFO:tasks.workunit.client.0.trial034.stdout:adding osd5 key to auth repository 2026-02-22T22:03:02.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:03:02.973 INFO:tasks.workunit.client.0.trial034.stdout:start osd.5 2026-02-22T22:03:02.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2026-02-22T22:03:02.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:03:02.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:03:02.975 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:03:02.975 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:03:02.995 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:02.992+0000 7fa10d6668c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:02.997 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:02.992+0000 7fa10d6668c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:02.999 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:02.996+0000 7fa10d6668c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:03.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2026-02-22T22:03:03.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:03:03.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2026-02-22T22:03:03.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:03:03.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:03:03.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:03.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:03:03.178 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:03:03.179 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:03.179 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T22:03:03.370 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:04.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:04.372 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:04.372 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:03:04.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:03:04.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:04.373 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T22:03:04.524 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:04.520+0000 7fa10d6668c0 -1 Falling back to public interface 2026-02-22T22:03:04.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:05.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:05.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:05.567 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:03:05.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:03:05.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:05.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T22:03:05.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:06.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:06.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:06.760 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:03:06.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:03:06.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:06.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T22:03:06.953 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:07.560 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:07.556+0000 7fa10d6668c0 -1 osd.5 0 log_to_monitors true 2026-02-22T22:03:07.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:07.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:07.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:03:07.955 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:03:07.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:07.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T22:03:08.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:09.150 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:09.150 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:09.150 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:03:09.150 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:03:09.151 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:09.151 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2026-02-22T22:03:09.342 INFO:tasks.workunit.client.0.trial034.stdout:osd.5 up in weight 1 up_from 43 up_thru 44 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/805786405,v1:127.0.0.1:6843/805786405] [v2:127.0.0.1:6844/805786405,v1:127.0.0.1:6845/805786405] exists,up 7a9492b7-2aba-4520-952a-e5fbb7659aa4 2026-02-22T22:03:09.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:03:09.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:03:09.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:03:09.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:03:09.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2026-02-22T22:03:09.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:03:09.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:03:09.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2026-02-22T22:03:09.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:03:09.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2026-02-22T22:03:09.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:03:09.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:03:09.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:03:09.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:03:09.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2026-02-22T22:03:09.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2026-02-22T22:03:09.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:03:09.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:03:09.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:03:09.347 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:03:09.347 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:03:09.347 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:03:09.348 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:03:09.348 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:09.348 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:09.348 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:03:09.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:03:09.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:03:09.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:03:09.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:03:09.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:03:09.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:03:09.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:03:09.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:03:09.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:03:09.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:03:09.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:03:09.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:03:09.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2026-02-22T22:03:09.349 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:03:09.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eacfa8cb-6d36-41bf-89f5-f7ce8bae2cfd 2026-02-22T22:03:09.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 eacfa8cb-6d36-41bf-89f5-f7ce8bae2cfd' 2026-02-22T22:03:09.350 INFO:tasks.workunit.client.0.trial034.stdout:add osd6 eacfa8cb-6d36-41bf-89f5-f7ce8bae2cfd 2026-02-22T22:03:09.350 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:03:09.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAdfZtpcDZlFRAAm4QH7vR13Dx7heB4Bua2tw== 2026-02-22T22:03:09.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAdfZtpcDZlFRAAm4QH7vR13Dx7heB4Bua2tw=="}' 2026-02-22T22:03:09.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eacfa8cb-6d36-41bf-89f5-f7ce8bae2cfd -i td/test-erasure-eio/6/new.json 2026-02-22T22:03:09.553 INFO:tasks.workunit.client.0.trial034.stdout:6 2026-02-22T22:03:09.562 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2026-02-22T22:03:09.563 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdfZtpcDZlFRAAm4QH7vR13Dx7heB4Bua2tw== --osd-uuid eacfa8cb-6d36-41bf-89f5-f7ce8bae2cfd 2026-02-22T22:03:09.588 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:09.584+0000 7fd3cfc6e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:09.589 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:09.584+0000 7fd3cfc6e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:09.590 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:09.588+0000 7fd3cfc6e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:09.591 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:09.588+0000 7fd3cfc6e8c0 -1 bdev(0x563c9ee7dc00 td/test-erasure-eio/6/block) open stat got: (1) Operation not permitted 2026-02-22T22:03:09.591 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:09.588+0000 7fd3cfc6e8c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2026-02-22T22:03:11.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2026-02-22T22:03:11.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:03:11.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2026-02-22T22:03:11.869 INFO:tasks.workunit.client.0.trial034.stdout:adding osd6 key to auth repository 2026-02-22T22:03:11.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:03:12.137 INFO:tasks.workunit.client.0.trial034.stdout:start osd.6 2026-02-22T22:03:12.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2026-02-22T22:03:12.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:03:12.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:03:12.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:03:12.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:03:12.159 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:12.156+0000 7f2c669888c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:12.160 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:12.156+0000 7f2c669888c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:12.161 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:12.156+0000 7f2c669888c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:12.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2026-02-22T22:03:12.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:03:12.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2026-02-22T22:03:12.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:03:12.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:03:12.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:12.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:03:12.340 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:03:12.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:12.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T22:03:12.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:13.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:13.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:13.541 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:03:13.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:03:13.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:13.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T22:03:13.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:13.900 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:13.896+0000 7f2c669888c0 -1 Falling back to public interface 2026-02-22T22:03:14.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:14.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:14.738 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:03:14.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:03:14.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:14.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T22:03:14.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:15.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:15.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:15.934 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:03:15.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:03:15.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:15.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T22:03:16.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:16.869 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:16.864+0000 7f2c669888c0 -1 osd.6 0 log_to_monitors true 2026-02-22T22:03:17.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:17.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:17.138 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:03:17.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:03:17.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:17.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T22:03:17.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:18.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:18.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:18.338 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:03:18.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:03:18.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:18.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2026-02-22T22:03:18.530 INFO:tasks.workunit.client.0.trial034.stdout:osd.6 up in weight 1 up_from 52 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/457170184,v1:127.0.0.1:6851/457170184] [v2:127.0.0.1:6852/457170184,v1:127.0.0.1:6853/457170184] exists,up eacfa8cb-6d36-41bf-89f5-f7ce8bae2cfd 2026-02-22T22:03:18.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:03:18.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:03:18.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:03:18.531 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T22:03:18.531 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:03:18.531 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:03:18.531 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:03:18.531 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:18.531 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:18.532 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T22:03:18.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T22:03:18.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T22:03:18.593 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:03:18.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T22:03:18.601 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:02:20.069+0000 7f2ab19658c0 0 load: jerasure load: lrc 2026-02-22T22:03:18.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2026-02-22T22:03:18.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:396: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2026-02-22T22:03:18.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:03:18.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T22:03:18.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2026-02-22T22:03:18.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T22:03:18.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2026-02-22T22:03:18.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T22:03:18.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2026-02-22T22:03:18.858 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T22:03:18.858 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T22:03:19.252 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T22:03:19.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:03:20.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T22:03:20.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:03:20.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:03:20.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:03:20.265 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:03:20.265 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:03:20.265 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:03:20.266 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:03:20.266 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:03:20.266 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:03:20.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:03:20.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:03:20.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:03:20.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:03:20.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:03:20.374 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:03:20.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:03:20.573 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:03:20.573 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:03:20.573 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:03:20.573 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:03:20.573 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T22:03:20.573 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T22:03:20.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:03:20.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:20.574 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:03:20.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803790 2026-02-22T22:03:20.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803790 2026-02-22T22:03:20.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790' 2026-02-22T22:03:20.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:20.648 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:03:20.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607564 2026-02-22T22:03:20.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607564 2026-02-22T22:03:20.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564' 2026-02-22T22:03:20.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:20.728 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:03:20.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411338 2026-02-22T22:03:20.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411338 2026-02-22T22:03:20.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338' 2026-02-22T22:03:20.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:20.802 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:03:20.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182409 2026-02-22T22:03:20.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182409 2026-02-22T22:03:20.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338 3-107374182409' 2026-02-22T22:03:20.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:20.877 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T22:03:20.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=146028888071 2026-02-22T22:03:20.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 146028888071 2026-02-22T22:03:20.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338 3-107374182409 4-146028888071' 2026-02-22T22:03:20.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:20.953 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T22:03:21.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=184683593733 2026-02-22T22:03:21.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 184683593733 2026-02-22T22:03:21.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338 3-107374182409 4-146028888071 5-184683593733' 2026-02-22T22:03:21.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:21.026 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T22:03:21.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=223338299395 2026-02-22T22:03:21.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 223338299395 2026-02-22T22:03:21.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803790 1-51539607564 2-77309411338 3-107374182409 4-146028888071 5-184683593733 6-223338299395' 2026-02-22T22:03:21.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:21.101 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803790 2026-02-22T22:03:21.102 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:21.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:03:21.103 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803790 2026-02-22T22:03:21.103 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:21.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803790 2026-02-22T22:03:21.105 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803790 2026-02-22T22:03:21.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803790' 2026-02-22T22:03:21.106 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:03:21.309 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803788 -lt 25769803790 2026-02-22T22:03:21.309 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:03:22.310 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:03:22.310 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:03:22.516 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803790 -lt 25769803790 2026-02-22T22:03:22.516 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:22.517 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607564 2026-02-22T22:03:22.517 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:22.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:03:22.519 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607564 2026-02-22T22:03:22.519 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:22.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607564 2026-02-22T22:03:22.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607564' 2026-02-22T22:03:22.521 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607564 2026-02-22T22:03:22.521 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:03:22.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607564 -lt 51539607564 2026-02-22T22:03:22.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:22.719 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411338 2026-02-22T22:03:22.719 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:22.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:03:22.721 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411338 2026-02-22T22:03:22.721 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:22.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411338 2026-02-22T22:03:22.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411338' 2026-02-22T22:03:22.723 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 77309411338 2026-02-22T22:03:22.723 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:03:22.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411338 -lt 77309411338 2026-02-22T22:03:22.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:22.925 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182409 2026-02-22T22:03:22.925 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:22.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:03:22.927 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182409 2026-02-22T22:03:22.927 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:22.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182409 2026-02-22T22:03:22.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182409' 2026-02-22T22:03:22.929 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 107374182409 2026-02-22T22:03:22.929 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:03:23.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182410 -lt 107374182409 2026-02-22T22:03:23.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:23.138 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-146028888071 2026-02-22T22:03:23.139 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:23.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T22:03:23.140 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-146028888071 2026-02-22T22:03:23.140 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:23.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=146028888071 2026-02-22T22:03:23.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 146028888071' 2026-02-22T22:03:23.142 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 146028888071 2026-02-22T22:03:23.142 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T22:03:23.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 146028888071 -lt 146028888071 2026-02-22T22:03:23.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:23.340 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-184683593733 2026-02-22T22:03:23.340 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:23.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T22:03:23.342 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-184683593733 2026-02-22T22:03:23.342 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:23.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=184683593733 2026-02-22T22:03:23.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 184683593733' 2026-02-22T22:03:23.344 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 184683593733 2026-02-22T22:03:23.345 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T22:03:23.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 184683593733 -lt 184683593733 2026-02-22T22:03:23.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:23.542 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 6-223338299395 2026-02-22T22:03:23.542 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:23.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=6 2026-02-22T22:03:23.544 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 6-223338299395 2026-02-22T22:03:23.544 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:23.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=223338299395 2026-02-22T22:03:23.545 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.6 seq 223338299395 2026-02-22T22:03:23.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.6 seq 223338299395' 2026-02-22T22:03:23.546 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T22:03:23.754 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 223338299395 -lt 223338299395 2026-02-22T22:03:23.754 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:03:23.755 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:03:23.755 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:03:24.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:03:24.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:03:24.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:03:24.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:03:24.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:03:24.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:03:24.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:03:24.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:03:24.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:03:24.212 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:03:24.212 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:03:24.212 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:03:24.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2026-02-22T22:03:24.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:399: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=myobject 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=0 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure myobject 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-22T22:03:24.494 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:03:24.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:03:24.688 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T22:03:24.688 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:03:24.688 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:03:24.688 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T22:03:24.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2026-02-22T22:03:24.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2026-02-22T22:03:24.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T22:03:24.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=3 2026-02-22T22:03:24.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T22:03:24.689 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/3/type 2026-02-22T22:03:24.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T22:03:24.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-22T22:03:24.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T22:03:24.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2026-02-22T22:03:24.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T22:03:24.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T22:03:24.690 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T22:03:24.690 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2026-02-22T22:03:24.691 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T22:03:24.691 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T22:03:24.691 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:03:24.691 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:24.691 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:24.691 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T22:03:24.692 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-22T22:03:24.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T22:03:24.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T22:03:24.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T22:03:24.753 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.3 2026-02-22T22:03:24.753 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T22:03:24.753 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T22:03:24.754 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:03:24.755 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:24.755 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:24.755 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T22:03:24.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T22:03:24.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2026-02-22T22:03:24.813 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2026-02-22T22:03:24.813 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:03:24.813 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2026-02-22T22:03:24.814 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2026-02-22T22:03:24.814 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:03:25.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:03:25.011 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T22:03:25.011 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:03:25.011 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:03:25.011 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T22:03:25.012 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2026-02-22T22:03:25.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: initial_osds=('3' '5' '1' '0' '6') 2026-02-22T22:03:25.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: local -a initial_osds 2026-02-22T22:03:25.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:402: TEST_ec_single_recovery_error: local last_osd=6 2026-02-22T22:03:25.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:404: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2026-02-22T22:03:25.012 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:03:25.012 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:03:25.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:03:25.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:03:25.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:03:26.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:03:26.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_single_recovery_error: ceph osd down 6 2026-02-22T22:03:26.509 INFO:tasks.workunit.client.0.trial034.stderr:osd.6 is already down. 2026-02-22T22:03:26.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:406: TEST_ec_single_recovery_error: ceph osd out 6 2026-02-22T22:03:26.732 INFO:tasks.workunit.client.0.trial034.stderr:osd.6 is already out. 2026-02-22T22:03:26.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:409: TEST_ec_single_recovery_error: wait_for_clean 2026-02-22T22:03:26.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:03:26.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:03:26.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:03:26.742 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:03:26.743 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:03:26.743 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:03:26.743 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:03:26.743 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:03:26.743 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:03:26.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:03:26.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:03:26.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:03:26.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:03:26.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:03:26.853 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:03:27.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:03:27.054 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:03:27.054 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:03:27.054 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:03:27.055 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:03:27.055 INFO:tasks.workunit.client.0.trial034.stderr:5 2026-02-22T22:03:27.055 INFO:tasks.workunit.client.0.trial034.stderr:6' 2026-02-22T22:03:27.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:03:27.055 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:27.055 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:03:27.130 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803793 2026-02-22T22:03:27.130 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803793 2026-02-22T22:03:27.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793' 2026-02-22T22:03:27.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:27.131 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:03:27.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607567 2026-02-22T22:03:27.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607567 2026-02-22T22:03:27.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567' 2026-02-22T22:03:27.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:27.206 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:03:27.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411341 2026-02-22T22:03:27.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411341 2026-02-22T22:03:27.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-77309411341' 2026-02-22T22:03:27.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:27.281 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:03:27.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182413 2026-02-22T22:03:27.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182413 2026-02-22T22:03:27.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-77309411341 3-107374182413' 2026-02-22T22:03:27.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:27.360 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T22:03:27.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=146028888074 2026-02-22T22:03:27.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 146028888074 2026-02-22T22:03:27.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-77309411341 3-107374182413 4-146028888074' 2026-02-22T22:03:27.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:27.437 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T22:03:27.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=184683593736 2026-02-22T22:03:27.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 184683593736 2026-02-22T22:03:27.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-51539607567 2-77309411341 3-107374182413 4-146028888074 5-184683593736' 2026-02-22T22:03:27.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:03:27.514 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T22:03:27.569 INFO:tasks.workunit.client.0.trial034.stderr:Error ENXIO: problem getting command descriptions from osd.6 2026-02-22T22:03:27.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq= 2026-02-22T22:03:27.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z '' 2026-02-22T22:03:27.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: continue 2026-02-22T22:03:27.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:27.572 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803793 2026-02-22T22:03:27.572 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:27.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:03:27.574 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803793 2026-02-22T22:03:27.574 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:27.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803793 2026-02-22T22:03:27.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803793' 2026-02-22T22:03:27.576 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803793 2026-02-22T22:03:27.576 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:03:27.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803791 -lt 25769803793 2026-02-22T22:03:27.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:03:28.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:03:28.777 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:03:28.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803793 -lt 25769803793 2026-02-22T22:03:28.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:28.980 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607567 2026-02-22T22:03:28.980 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:28.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:03:28.982 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607567 2026-02-22T22:03:28.982 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:28.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607567 2026-02-22T22:03:28.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607567' 2026-02-22T22:03:28.984 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607567 2026-02-22T22:03:28.984 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:03:29.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607567 -lt 51539607567 2026-02-22T22:03:29.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:29.191 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411341 2026-02-22T22:03:29.192 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:29.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:03:29.194 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411341 2026-02-22T22:03:29.194 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:29.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411341 2026-02-22T22:03:29.195 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 77309411341 2026-02-22T22:03:29.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411341' 2026-02-22T22:03:29.196 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:03:29.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411342 -lt 77309411341 2026-02-22T22:03:29.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:29.395 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182413 2026-02-22T22:03:29.395 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:29.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:03:29.397 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182413 2026-02-22T22:03:29.398 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:29.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182413 2026-02-22T22:03:29.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182413' 2026-02-22T22:03:29.399 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 107374182413 2026-02-22T22:03:29.400 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:03:29.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182413 -lt 107374182413 2026-02-22T22:03:29.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:29.604 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 4-146028888074 2026-02-22T22:03:29.604 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:29.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=4 2026-02-22T22:03:29.606 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 4-146028888074 2026-02-22T22:03:29.606 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:29.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=146028888074 2026-02-22T22:03:29.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.4 seq 146028888074' 2026-02-22T22:03:29.608 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.4 seq 146028888074 2026-02-22T22:03:29.608 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T22:03:29.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 146028888074 -lt 146028888074 2026-02-22T22:03:29.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:03:29.809 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 5-184683593736 2026-02-22T22:03:29.809 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:03:29.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=5 2026-02-22T22:03:29.811 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 5-184683593736 2026-02-22T22:03:29.811 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:03:29.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=184683593736 2026-02-22T22:03:29.813 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.5 seq 184683593736 2026-02-22T22:03:29.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.5 seq 184683593736' 2026-02-22T22:03:29.813 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T22:03:30.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 184683593736 -lt 184683593736 2026-02-22T22:03:30.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:03:30.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:03:30.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:03:30.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:03:30.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:03:30.271 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:03:30.271 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:03:30.271 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:03:30.271 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:03:30.271 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:03:30.271 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:03:30.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:03:30.465 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:03:30.466 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:03:30.466 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:03:30.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 5 2026-02-22T22:03:30.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 4 '!=' -1 2026-02-22T22:03:30.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: loop=0 2026-02-22T22:03:30.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: num_active_clean=4 2026-02-22T22:03:30.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:03:30.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.1 2026-02-22T22:03:30.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:03:30.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:03:30.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:03:30.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:03:30.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:03:30.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:03:30.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:03:30.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:03:31.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:03:31.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:03:31.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:03:31.012 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:03:31.264 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 5 2026-02-22T22:03:31.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 4 '!=' 4 2026-02-22T22:03:31.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:03:31.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:03:31.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:03:31.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:03:31.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:03:31.265 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:03:31.265 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:03:31.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2026-02-22T22:03:31.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2026-02-22T22:03:31.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: (( 1 >= 13 )) 2026-02-22T22:03:31.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:03:31.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.2 2026-02-22T22:03:31.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:03:31.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:03:31.718 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:03:31.718 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:03:31.718 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:03:31.718 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:03:31.718 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:03:31.718 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:03:31.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=4 2026-02-22T22:03:31.911 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:03:31.911 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:03:31.911 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:03:32.173 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 4 = 5 2026-02-22T22:03:32.173 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: test 4 '!=' 4 2026-02-22T22:03:32.174 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: get_is_making_recovery_progress 2026-02-22T22:03:32.174 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2026-02-22T22:03:32.174 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T22:03:32.174 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T22:03:32.174 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T22:03:32.174 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2026-02-22T22:03:32.174 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T22:03:32.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1023 2026-02-22T22:03:32.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1023 '!=' null 2026-02-22T22:03:32.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: wait_for_clean: loop=0 2026-02-22T22:03:32.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: wait_for_clean: eval 2026-02-22T22:03:32.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: wait_for_clean: sleep 0.1 2026-02-22T22:03:32.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: wait_for_clean: loop+=1 2026-02-22T22:03:32.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:03:32.535 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:03:32.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:03:32.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:03:32.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:03:32.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:03:32.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:03:32.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:03:32.731 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:03:32.731 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:03:32.731 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:03:32.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:03:32.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:03:32.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:03:32.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2026-02-22T22:03:32.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:03:32.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:03:32.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2026-02-22T22:03:32.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:03:32.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:03:32.992 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2026-02-22T22:03:33.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:03:33.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:03:33.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2026-02-22T22:03:33.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:03:33.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T22:03:33.300 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T22:03:33.309 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T22:03:33.508 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T22:03:33.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T22:03:33.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:03:33.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:03:33.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:03:33.517 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:03:33.517 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:03:33.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:03:33.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:03:33.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:03:33.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:03:33.830 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:03:33.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:03:33.832 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:03:33.834 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:03:33.834 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:03:33.834 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:03:33.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:03:33.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:03:33.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:03:33.836 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:03:33.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:03:33.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:03:33.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:03:33.863 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:03:33.863 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:33.863 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:33.863 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:03:33.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:03:33.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:03:33.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T22:03:33.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T22:03:33.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T22:03:33.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T22:03:33.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:03:33.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:03:33.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:03:33.865 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:03:33.865 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:03:33.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:03:33.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:03:33.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:03:33.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:03:33.868 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:03:33.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:03:33.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:03:33.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:03:33.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:03:33.871 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:03:33.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:03:33.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:03:33.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:03:33.873 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:03:33.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:03:33.875 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:03:33.875 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:03:33.876 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:03:33.876 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:33.876 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:33.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:03:33.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:03:33.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:03:33.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T22:03:33.879 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:03:33.879 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:33.879 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:33.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T22:03:33.881 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:03:33.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:03:33.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:03:33.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T22:03:33.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T22:03:33.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T22:03:33.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:03:33.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:03:33.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:03:33.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T22:03:33.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T22:03:33.904 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:03:33.904 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:03:33.904 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:03:33.904 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:03:33.904 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:33.904 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:33.904 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:03:33.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:03:33.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:03:33.934 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:03:33.934 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:03:33.934 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:03:33.934 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:03:33.935 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:03:33.935 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:03:33.935 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:03:33.935 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:03:33.935 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:03:33.936 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:33.936 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:33.936 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:03:33.936 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:03:33.936 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T22:03:33.992 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:03:33.993 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:03:33.993 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:03:33.993 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:03:33.993 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:03:33.994 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:03:33.994 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:03:33.995 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:03:33.995 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:03:33.995 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:33.995 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:33.995 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:03:33.995 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:03:33.995 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T22:03:34.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T22:03:34.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T22:03:34.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:03:34.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:03:34.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:03:34.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T22:03:34.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:03:34.156 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:03:34.156 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:03:34.156 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:03:34.157 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:03:34.157 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:34.157 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:34.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:03:34.158 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:03:34.159 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:03:34.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T22:03:34.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:03:34.285 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T22:03:34.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:03:35.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T22:03:35.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:03:35.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:03:35.295 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:03:35.295 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:35.295 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:35.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:03:35.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T22:03:35.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T22:03:35.345 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:03:35.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T22:03:35.352 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:03:33.924+0000 7f61212a4ec0 0 load: jerasure load: lrc 2026-02-22T22:03:35.353 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2026-02-22T22:03:35.353 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2026-02-22T22:03:35.353 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_bad_size_shard_0: setup_osds 4 2026-02-22T22:03:35.353 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-22T22:03:35.353 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T22:03:35.353 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-22T22:03:35.354 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-22T22:03:35.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:03:35.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T22:03:35.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:03:35.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:03:35.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:03:35.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:03:35.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T22:03:35.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:03:35.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:03:35.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:03:35.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:03:35.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:03:35.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:03:35.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:03:35.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:03:35.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:03:35.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:03:35.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:03:35.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:03:35.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T22:03:35.360 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:03:35.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a2640d18-1f60-4e04-89af-84c2baed2db3 2026-02-22T22:03:35.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a2640d18-1f60-4e04-89af-84c2baed2db3' 2026-02-22T22:03:35.361 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 a2640d18-1f60-4e04-89af-84c2baed2db3 2026-02-22T22:03:35.361 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:03:35.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3fZtpetguFhAAnPQB8pe7ehwNiqaI+oPuiA== 2026-02-22T22:03:35.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3fZtpetguFhAAnPQB8pe7ehwNiqaI+oPuiA=="}' 2026-02-22T22:03:35.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a2640d18-1f60-4e04-89af-84c2baed2db3 -i td/test-erasure-eio/0/new.json 2026-02-22T22:03:35.479 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:03:35.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T22:03:35.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3fZtpetguFhAAnPQB8pe7ehwNiqaI+oPuiA== --osd-uuid a2640d18-1f60-4e04-89af-84c2baed2db3 2026-02-22T22:03:35.504 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:35.504+0000 7f90b52f78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:35.505 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:35.504+0000 7f90b52f78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:35.506 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:35.508+0000 7f90b52f78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:35.506 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:35.508+0000 7f90b52f78c0 -1 bdev(0x561ccf273c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:03:35.507 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:35.508+0000 7f90b52f78c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T22:03:38.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T22:03:38.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:03:38.068 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T22:03:38.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:03:38.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:03:38.346 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T22:03:38.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:03:38.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:03:38.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:03:38.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:03:38.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:03:38.370 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:38.367+0000 7fc1c1cb28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:38.371 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:38.371+0000 7fc1c1cb28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:38.373 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:38.371+0000 7fc1c1cb28c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:38.538 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:03:38.538 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:03:38.538 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:03:38.538 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:03:38.538 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:03:38.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:38.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:03:38.539 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:03:38.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:38.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:03:38.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:39.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:39.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:39.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:03:39.742 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:03:39.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:39.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:03:39.880 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:39.879+0000 7fc1c1cb28c0 -1 Falling back to public interface 2026-02-22T22:03:39.941 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:40.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:40.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:40.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:03:40.942 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:03:40.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:40.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:03:41.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:42.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:42.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:42.143 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:03:42.143 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:03:42.144 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:42.144 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:03:42.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:43.098 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:43.099+0000 7fc1c1cb28c0 -1 osd.0 0 log_to_monitors true 2026-02-22T22:03:43.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:43.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:43.347 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:03:43.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:03:43.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:43.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:03:43.549 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:44.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:44.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:44.550 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:03:44.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:03:44.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:44.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:03:44.753 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1588954399,v1:127.0.0.1:6803/1588954399] [v2:127.0.0.1:6804/1588954399,v1:127.0.0.1:6805/1588954399] exists,up a2640d18-1f60-4e04-89af-84c2baed2db3 2026-02-22T22:03:44.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:03:44.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:03:44.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:03:44.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:03:44.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T22:03:44.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:03:44.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:03:44.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:03:44.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:03:44.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:03:44.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:03:44.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:03:44.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:03:44.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:03:44.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:03:44.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:03:44.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:03:44.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:03:44.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:03:44.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:03:44.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:03:44.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:03:44.758 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:03:44.759 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:44.759 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:44.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:03:44.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:03:44.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:03:44.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:03:44.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:03:44.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:03:44.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:03:44.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:03:44.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:03:44.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:03:44.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:03:44.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:03:44.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:03:44.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:03:44.760 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:03:44.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=52a45d98-de07-474d-b7d4-7b98909dbf26 2026-02-22T22:03:44.761 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 52a45d98-de07-474d-b7d4-7b98909dbf26 2026-02-22T22:03:44.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 52a45d98-de07-474d-b7d4-7b98909dbf26' 2026-02-22T22:03:44.761 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:03:44.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAfZtpLO3hLRAAggS09AmpJmVbnj2vgmhQLA== 2026-02-22T22:03:44.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAfZtpLO3hLRAAggS09AmpJmVbnj2vgmhQLA=="}' 2026-02-22T22:03:44.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 52a45d98-de07-474d-b7d4-7b98909dbf26 -i td/test-erasure-eio/1/new.json 2026-02-22T22:03:44.971 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:03:44.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T22:03:44.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAfZtpLO3hLRAAggS09AmpJmVbnj2vgmhQLA== --osd-uuid 52a45d98-de07-474d-b7d4-7b98909dbf26 2026-02-22T22:03:44.997 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:44.995+0000 7f25688608c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:44.998 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:44.999+0000 7f25688608c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:44.999 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:44.999+0000 7f25688608c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:45.000 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:44.999+0000 7f25688608c0 -1 bdev(0x564ec5dddc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:03:45.000 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:44.999+0000 7f25688608c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T22:03:47.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T22:03:47.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:03:47.281 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T22:03:47.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:03:47.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:03:47.553 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T22:03:47.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:03:47.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:03:47.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:03:47.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:03:47.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:03:47.575 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:47.575+0000 7f24014448c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:47.576 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:47.575+0000 7f24014448c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:47.577 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:47.575+0000 7f24014448c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:47.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:03:47.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:03:47.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:03:47.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:03:47.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:03:47.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:47.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:03:47.757 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:03:47.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:47.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:03:47.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:48.953 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:48.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:48.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:03:48.954 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:03:48.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:48.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:03:49.153 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:49.292 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:49.291+0000 7f24014448c0 -1 Falling back to public interface 2026-02-22T22:03:50.155 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:50.155 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:50.155 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:03:50.156 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:03:50.156 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:50.156 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:03:50.353 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:51.355 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:51.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:51.356 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:03:51.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:03:51.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:51.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:03:51.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:52.311 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:52.311+0000 7f24014448c0 -1 osd.1 0 log_to_monitors true 2026-02-22T22:03:52.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:52.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:52.554 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:03:52.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:03:52.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:52.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:03:52.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:53.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:53.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:53.749 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:03:53.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:03:53.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:53.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:03:53.951 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2835697387,v1:127.0.0.1:6811/2835697387] [v2:127.0.0.1:6812/2835697387,v1:127.0.0.1:6813/2835697387] exists,up 52a45d98-de07-474d-b7d4-7b98909dbf26 2026-02-22T22:03:53.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:03:53.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:03:53.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:03:53.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:03:53.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T22:03:53.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:03:53.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:03:53.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:03:53.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:03:53.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:03:53.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:03:53.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:03:53.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:03:53.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:03:53.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:03:53.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:03:53.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:03:53.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:03:53.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:03:53.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:03:53.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:03:53.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:03:53.956 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:03:53.956 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:03:53.956 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:03:53.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:03:53.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:03:53.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:03:53.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:03:53.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:03:53.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:03:53.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:03:53.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:03:53.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:03:53.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:03:53.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:03:53.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:03:53.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:03:53.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:03:53.958 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:03:53.958 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b6e8b2a5-9162-436e-9897-351af88e98fc 2026-02-22T22:03:53.958 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b6e8b2a5-9162-436e-9897-351af88e98fc' 2026-02-22T22:03:53.959 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 b6e8b2a5-9162-436e-9897-351af88e98fc 2026-02-22T22:03:53.959 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:03:53.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBJfZtpaSa/ORAAoPToWKEqyzcm1CCcONCmHw== 2026-02-22T22:03:53.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBJfZtpaSa/ORAAoPToWKEqyzcm1CCcONCmHw=="}' 2026-02-22T22:03:53.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b6e8b2a5-9162-436e-9897-351af88e98fc -i td/test-erasure-eio/2/new.json 2026-02-22T22:03:54.177 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:03:54.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T22:03:54.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJfZtpaSa/ORAAoPToWKEqyzcm1CCcONCmHw== --osd-uuid b6e8b2a5-9162-436e-9897-351af88e98fc 2026-02-22T22:03:54.204 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:54.203+0000 7f02a6ebe8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:54.205 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:54.203+0000 7f02a6ebe8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:54.207 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:54.207+0000 7f02a6ebe8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:54.207 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:54.207+0000 7f02a6ebe8c0 -1 bdev(0x562943db9c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:03:54.207 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:54.207+0000 7f02a6ebe8c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T22:03:56.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T22:03:56.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:03:56.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:03:56.515 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T22:03:56.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:03:56.785 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T22:03:56.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:03:56.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:03:56.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:03:56.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:03:56.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:03:56.809 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:56.807+0000 7f420c9998c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:56.811 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:56.811+0000 7f420c9998c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:56.812 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:56.811+0000 7f420c9998c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:03:56.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:03:56.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:03:56.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:03:56.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:03:56.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:03:56.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:56.981 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:03:56.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:03:56.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:56.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:03:57.183 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:58.184 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:58.184 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:58.184 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:03:58.184 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:03:58.185 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:58.185 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:03:58.312 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:03:58.311+0000 7f420c9998c0 -1 Falling back to public interface 2026-02-22T22:03:58.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:03:59.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:03:59.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:03:59.392 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:03:59.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:03:59.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:03:59.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:03:59.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:00.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:00.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:00.592 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:04:00.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:04:00.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:00.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:04:00.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:01.585 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:01.583+0000 7f420c9998c0 -1 osd.2 0 log_to_monitors true 2026-02-22T22:04:01.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:01.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:01.794 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:04:01.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:04:01.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:01.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:04:02.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:03.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:03.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:03.007 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:04:03.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:04:03.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:03.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:04:03.206 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 18 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4036038331,v1:127.0.0.1:6819/4036038331] [v2:127.0.0.1:6820/4036038331,v1:127.0.0.1:6821/4036038331] exists,up b6e8b2a5-9162-436e-9897-351af88e98fc 2026-02-22T22:04:03.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:04:03.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:04:03.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:04:03.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:04:03.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T22:04:03.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:04:03.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:04:03.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T22:04:03.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:04:03.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:04:03.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:04:03.211 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:04:03.212 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:04:03.212 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:04:03.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:04:03.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:04:03.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:04:03.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:04:03.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:04:03.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:04:03.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:04:03.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:04:03.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:04:03.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:04:03.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:04:03.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:04:03.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:04:03.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:04:03.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7cfc75d6-c7bf-4610-ab5b-211726921771 2026-02-22T22:04:03.214 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 7cfc75d6-c7bf-4610-ab5b-211726921771 2026-02-22T22:04:03.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 7cfc75d6-c7bf-4610-ab5b-211726921771' 2026-02-22T22:04:03.214 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:04:03.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTfZtpAHNJDRAAMcPrjgPcZqkG3o+AkOlOKw== 2026-02-22T22:04:03.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTfZtpAHNJDRAAMcPrjgPcZqkG3o+AkOlOKw=="}' 2026-02-22T22:04:03.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7cfc75d6-c7bf-4610-ab5b-211726921771 -i td/test-erasure-eio/3/new.json 2026-02-22T22:04:03.434 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:04:03.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T22:04:03.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTfZtpAHNJDRAAMcPrjgPcZqkG3o+AkOlOKw== --osd-uuid 7cfc75d6-c7bf-4610-ab5b-211726921771 2026-02-22T22:04:03.459 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:03.459+0000 7fecefa438c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:03.460 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:03.459+0000 7fecefa438c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:03.461 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:03.459+0000 7fecefa438c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:03.461 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:03.459+0000 7fecefa438c0 -1 bdev(0x5559c217fc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T22:04:03.462 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:03.459+0000 7fecefa438c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T22:04:05.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T22:04:05.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:04:05.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T22:04:05.749 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T22:04:05.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:04:06.021 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T22:04:06.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T22:04:06.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:04:06.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:04:06.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:04:06.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:04:06.044 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:06.043+0000 7ff1c24f58c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:06.046 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:06.043+0000 7ff1c24f58c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:06.047 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:06.047+0000 7ff1c24f58c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:06.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T22:04:06.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:04:06.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:04:06.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:04:06.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:04:06.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:06.222 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:04:06.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:04:06.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:06.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:06.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:07.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:07.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:07.424 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:04:07.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:04:07.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:07.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:07.544 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:07.543+0000 7ff1c24f58c0 -1 Falling back to public interface 2026-02-22T22:04:07.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:08.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:08.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:08.624 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:04:08.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:04:08.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:08.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:08.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:09.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:09.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:09.823 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:04:09.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:04:09.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:09.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:10.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:10.794 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:10.795+0000 7ff1c24f58c0 -1 osd.3 0 log_to_monitors true 2026-02-22T22:04:11.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:11.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:11.019 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:04:11.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:04:11.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:11.019 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:11.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:12.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:12.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:12.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:04:12.235 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:04:12.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:12.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:12.433 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 26 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2778219598,v1:127.0.0.1:6827/2778219598] [v2:127.0.0.1:6828/2778219598,v1:127.0.0.1:6829/2778219598] exists,up 7cfc75d6-c7bf-4610-ab5b-211726921771 2026-02-22T22:04:12.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:04:12.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:04:12.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:04:12.435 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T22:04:12.435 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:04:12.435 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:04:12.435 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:04:12.435 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:04:12.435 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:04:12.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T22:04:12.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T22:04:12.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T22:04:12.489 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:04:12.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T22:04:12.496 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:03:39.883+0000 7fc1c1cb28c0 0 load: jerasure load: lrc 2026-02-22T22:04:12.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2026-02-22T22:04:12.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2026-02-22T22:04:12.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:04:12.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T22:04:12.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-22T22:04:12.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T22:04:12.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-22T22:04:12.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T22:04:12.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-22T22:04:12.753 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T22:04:12.754 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T22:04:13.108 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T22:04:13.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:04:14.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T22:04:14.119 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:04:14.119 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:04:14.119 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:04:14.119 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:04:14.120 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:04:14.120 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:04:14.120 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:04:14.121 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:04:14.121 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:04:14.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:04:14.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:04:14.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:04:14.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:04:14.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:04:14.232 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:04:14.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:04:14.428 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:04:14.428 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:04:14.428 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:04:14.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:04:14.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:14.429 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:04:14.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803785 2026-02-22T22:04:14.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803785 2026-02-22T22:04:14.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803785' 2026-02-22T22:04:14.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:14.505 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:04:14.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=55834574855 2026-02-22T22:04:14.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 55834574855 2026-02-22T22:04:14.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803785 1-55834574855' 2026-02-22T22:04:14.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:14.580 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:04:14.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411333 2026-02-22T22:04:14.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411333 2026-02-22T22:04:14.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803785 1-55834574855 2-77309411333' 2026-02-22T22:04:14.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:14.656 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:04:14.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=111669149699 2026-02-22T22:04:14.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 111669149699 2026-02-22T22:04:14.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803785 1-55834574855 2-77309411333 3-111669149699' 2026-02-22T22:04:14.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:14.730 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803785 2026-02-22T22:04:14.730 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:14.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:04:14.732 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803785 2026-02-22T22:04:14.732 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:14.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803785 2026-02-22T22:04:14.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2026-02-22T22:04:14.734 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803785 2026-02-22T22:04:14.734 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:04:14.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803783 -lt 25769803785 2026-02-22T22:04:14.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:04:15.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:04:15.941 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:04:16.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803783 -lt 25769803785 2026-02-22T22:04:16.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:04:17.137 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:04:17.137 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:04:17.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803785 -lt 25769803785 2026-02-22T22:04:17.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:17.338 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-55834574855 2026-02-22T22:04:17.338 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:17.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:04:17.340 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-55834574855 2026-02-22T22:04:17.340 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:17.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=55834574855 2026-02-22T22:04:17.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 55834574855' 2026-02-22T22:04:17.342 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 55834574855 2026-02-22T22:04:17.342 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:04:17.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 55834574855 -lt 55834574855 2026-02-22T22:04:17.539 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:17.540 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411333 2026-02-22T22:04:17.540 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:17.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:04:17.542 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411333 2026-02-22T22:04:17.542 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:17.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411333 2026-02-22T22:04:17.544 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 77309411333 2026-02-22T22:04:17.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411333' 2026-02-22T22:04:17.544 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:04:17.735 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411333 -lt 77309411333 2026-02-22T22:04:17.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:17.736 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-111669149699 2026-02-22T22:04:17.736 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:17.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:04:17.738 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-111669149699 2026-02-22T22:04:17.738 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:17.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=111669149699 2026-02-22T22:04:17.739 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 111669149699 2026-02-22T22:04:17.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 111669149699' 2026-02-22T22:04:17.740 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:04:17.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 111669149699 -lt 111669149699 2026-02-22T22:04:17.932 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:04:17.933 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:04:17.933 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:04:18.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:04:18.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:04:18.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:04:18.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:04:18.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:04:18.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:04:18.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:04:18.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:04:18.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:04:18.392 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:04:18.393 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:04:18.393 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:04:18.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:04:18.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:04:18.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:04:18.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_bad_size_shard_0: local shard_id=0 2026-02-22T22:04:18.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2026-02-22T22:04:18.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-22T22:04:18.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2026-02-22T22:04:18.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2026-02-22T22:04:18.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2026-02-22T22:04:18.660 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2026-02-22T22:04:18.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2026-02-22T22:04:18.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2026-02-22T22:04:18.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-22T22:04:18.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-71671-0-10 2026-02-22T22:04:18.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-71671-0-10 2026-02-22T22:04:18.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:04:18.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:04:18.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-71671-0-10 2026-02-22T22:04:18.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:04:18.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:04:18.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:04:18.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:04:18.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:04:18.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:04:18.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:04:18.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:04:18.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-71671-0-10 td/test-erasure-eio/ORIGINAL 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-71671-0-10 td/test-erasure-eio 0 10 set 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-71671-0-10 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-71671-0-10 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:04:18.676 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-71671-0-10 2026-02-22T22:04:18.677 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-71671-0-10 2026-02-22T22:04:18.677 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:04:18.871 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:04:18.871 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:04:18.871 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:04:18.871 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:04:18.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2026-02-22T22:04:18.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2026-02-22T22:04:18.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2026-02-22T22:04:18.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2026-02-22T22:04:19.176 INFO:tasks.workunit.client.0.trial034.stderr:noout is set 2026-02-22T22:04:19.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2026-02-22T22:04:19.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2026-02-22T22:04:19.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records in 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records out 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:10 bytes copied, 3.0847e-05 s, 324 kB/s 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-71671-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-71671-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2026-02-22T22:04:19.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:04:19.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-22T22:04:19.189 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:04:19.189 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:04:19.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:04:19.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:04:19.189 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:04:20.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:04:20.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-71671-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:20.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:04:20.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:04:20.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2026-02-22T22:04:20.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:04:20.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2026-02-22T22:04:20.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-71671-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:04:21.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:04:21.685 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:04:21.685 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:04:21.685 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:04:21.685 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:04:21.685 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:04:21.685 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:04:21.686 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:04:21.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:04:21.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:04:21.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:04:21.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:04:21.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:04:21.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:04:21.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:04:21.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:04:21.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:04:21.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:04:21.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:04:21.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2026-02-22T22:04:21.687 INFO:tasks.workunit.client.0.trial034.stderr:start osd.3 2026-02-22T22:04:21.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:04:21.687 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-22T22:04:21.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2026-02-22T22:04:21.688 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:04:21.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:04:21.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:04:21.710 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:21.707+0000 7f74075e78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:21.711 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:21.711+0000 7f74075e78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:21.712 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:21.711+0000 7f74075e78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:21.886 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2026-02-22T22:04:21.886 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:04:21.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:04:21.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:04:21.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:04:21.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:21.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:04:21.887 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:04:21.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:21.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:22.085 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:23.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:23.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:23.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:04:23.086 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:04:23.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:23.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:23.192 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:23.191+0000 7f74075e78c0 -1 Falling back to public interface 2026-02-22T22:04:23.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:24.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:24.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:24.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:04:24.286 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:04:24.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:24.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:24.482 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:25.484 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:25.484 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:25.484 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:04:25.484 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:04:25.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:25.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:25.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:26.158 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:26.158+0000 7f74075e78c0 -1 osd.3 35 log_to_monitors true 2026-02-22T22:04:26.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:26.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:26.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:04:26.678 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:04:26.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:26.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:26.881 INFO:tasks.workunit.client.0.trial034.stderr:osd.3 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [26,35) [v2:127.0.0.1:6826/3910577427,v1:127.0.0.1:6827/3910577427] [v2:127.0.0.1:6828/3910577427,v1:127.0.0.1:6829/3910577427] exists,up 7cfc75d6-c7bf-4610-ab5b-211726921771 2026-02-22T22:04:26.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:04:26.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:04:26.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:04:26.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:04:26.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:04:26.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:04:26.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:04:26.883 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:04:26.883 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:04:26.883 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:04:26.883 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:04:26.884 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:04:26.884 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:04:26.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:04:26.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:04:26.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:04:26.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:04:26.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:04:26.989 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:04:27.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:04:27.193 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:04:27.193 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:04:27.193 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:04:27.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:04:27.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:27.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:04:27.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803789 2026-02-22T22:04:27.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803789 2026-02-22T22:04:27.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789' 2026-02-22T22:04:27.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:27.268 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:04:27.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=55834574859 2026-02-22T22:04:27.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 55834574859 2026-02-22T22:04:27.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-55834574859' 2026-02-22T22:04:27.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:27.343 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:04:27.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411337 2026-02-22T22:04:27.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411337 2026-02-22T22:04:27.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-55834574859 2-77309411337' 2026-02-22T22:04:27.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:27.420 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:04:27.495 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=171798691843 2026-02-22T22:04:27.495 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 171798691843 2026-02-22T22:04:27.495 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-55834574859 2-77309411337 3-171798691843' 2026-02-22T22:04:27.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:27.496 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803789 2026-02-22T22:04:27.496 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:27.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:04:27.498 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803789 2026-02-22T22:04:27.498 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:27.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803789 2026-02-22T22:04:27.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803789' 2026-02-22T22:04:27.499 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803789 2026-02-22T22:04:27.500 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:04:27.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803787 -lt 25769803789 2026-02-22T22:04:27.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:04:28.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:04:28.708 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:04:28.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803789 -lt 25769803789 2026-02-22T22:04:28.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:28.906 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-55834574859 2026-02-22T22:04:28.906 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:28.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:04:28.908 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-55834574859 2026-02-22T22:04:28.908 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:28.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=55834574859 2026-02-22T22:04:28.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 55834574859' 2026-02-22T22:04:28.910 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 55834574859 2026-02-22T22:04:28.910 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:04:29.112 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 55834574859 -lt 55834574859 2026-02-22T22:04:29.112 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:29.112 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411337 2026-02-22T22:04:29.113 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:29.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:04:29.114 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411337 2026-02-22T22:04:29.114 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:29.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411337 2026-02-22T22:04:29.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411337' 2026-02-22T22:04:29.116 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 77309411337 2026-02-22T22:04:29.116 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:04:29.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411338 -lt 77309411337 2026-02-22T22:04:29.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:29.322 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-171798691843 2026-02-22T22:04:29.322 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:29.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:04:29.324 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-171798691843 2026-02-22T22:04:29.324 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:29.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=171798691843 2026-02-22T22:04:29.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 171798691843' 2026-02-22T22:04:29.326 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 171798691843 2026-02-22T22:04:29.326 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:04:29.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 171798691843 -lt 171798691843 2026-02-22T22:04:29.539 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:04:29.539 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:04:29.539 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:04:29.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:04:29.796 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:04:29.798 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:04:29.798 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:04:29.798 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:04:29.798 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:04:29.798 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:04:29.798 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:04:29.989 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:04:29.989 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:04:29.990 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:04:29.990 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:04:30.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:04:30.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:04:30.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:04:30.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-22T22:04:30.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2026-02-22T22:04:30.513 INFO:tasks.workunit.client.0.trial034.stderr:noout is unset 2026-02-22T22:04:30.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-71671-0-10 2026-02-22T22:04:30.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:04:30.526 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:04:30.526 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-71671-0-10 2026-02-22T22:04:30.526 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:04:30.526 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:04:30.526 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-71671-0-10 td/test-erasure-eio/COPY 2026-02-22T22:04:30.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:04:30.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:04:30.553 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2026-02-22T22:04:30.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2026-02-22T22:04:30.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-71671-0-10 td/test-erasure-eio 1 10 set 2026-02-22T22:04:30.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-71671-0-10 2026-02-22T22:04:30.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2026-02-22T22:04:30.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2026-02-22T22:04:30.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2026-02-22T22:04:30.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2026-02-22T22:04:30.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2026-02-22T22:04:30.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2026-02-22T22:04:30.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2026-02-22T22:04:30.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2026-02-22T22:04:30.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2026-02-22T22:04:30.557 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-71671-0-10 2026-02-22T22:04:30.557 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:04:30.557 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-71671-0-10 2026-02-22T22:04:30.557 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-71671-0-10 2026-02-22T22:04:30.557 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:04:30.750 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:04:30.751 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:04:30.751 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:04:30.751 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:04:30.751 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2026-02-22T22:04:30.751 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2026-02-22T22:04:30.751 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2026-02-22T22:04:30.751 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2026-02-22T22:04:31.017 INFO:tasks.workunit.client.0.trial034.stderr:noout is set 2026-02-22T22:04:31.027 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2026-02-22T22:04:31.027 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2026-02-22T22:04:31.028 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2026-02-22T22:04:31.029 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records in 2026-02-22T22:04:31.029 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records out 2026-02-22T22:04:31.029 INFO:tasks.workunit.client.0.trial034.stderr:10 bytes copied, 3.0817e-05 s, 324 kB/s 2026-02-22T22:04:31.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-71671-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:31.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:04:31.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:04:31.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2026-02-22T22:04:31.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:04:31.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-71671-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:31.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:04:31.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:04:31.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2026-02-22T22:04:31.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:04:31.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-22T22:04:31.030 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:04:31.031 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:04:31.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:04:31.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:04:31.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:04:32.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:04:32.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-71671-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:32.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:04:32.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:04:32.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2026-02-22T22:04:32.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:04:32.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-22T22:04:32.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-71671-0-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:33.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-22T22:04:33.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:04:33.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:04:33.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2026-02-22T22:04:33.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:04:33.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:04:33.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:04:33.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:04:33.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:04:33.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:04:33.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:04:33.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:04:33.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:04:33.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:04:33.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:04:33.567 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:04:33.567 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:04:33.567 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:04:33.568 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:04:33.568 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:04:33.568 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:04:33.569 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:04:33.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:04:33.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:04:33.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:04:33.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:04:33.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:04:33.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:04:33.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:04:33.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:04:33.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:04:33.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:04:33.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:04:33.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2026-02-22T22:04:33.570 INFO:tasks.workunit.client.0.trial034.stderr:start osd.1 2026-02-22T22:04:33.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:04:33.571 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-22T22:04:33.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2026-02-22T22:04:33.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:04:33.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:04:33.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:04:33.591 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:33.590+0000 7f4c4255b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:33.592 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:33.590+0000 7f4c4255b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:33.593 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:33.590+0000 7f4c4255b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:33.772 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2026-02-22T22:04:33.772 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:04:33.772 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:04:33.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:04:33.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:04:33.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:33.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:04:33.773 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:04:33.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:33.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:04:33.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:34.856 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:34.854+0000 7f4c4255b8c0 -1 Falling back to public interface 2026-02-22T22:04:34.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:34.970 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:34.970 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:04:34.970 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:04:34.970 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:34.970 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:04:35.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:36.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:36.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:36.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:04:36.166 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:04:36.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:36.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:04:36.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:37.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:37.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:37.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:04:37.359 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:04:37.359 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:37.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:04:37.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:38.132 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:38.130+0000 7f4c4255b8c0 -1 osd.1 43 log_to_monitors true 2026-02-22T22:04:38.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:38.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:38.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:04:38.556 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:04:38.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:38.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:04:38.747 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 46 last_clean_interval [13,43) [v2:127.0.0.1:6810/1759278813,v1:127.0.0.1:6811/1759278813] [v2:127.0.0.1:6812/1759278813,v1:127.0.0.1:6813/1759278813] exists,up 52a45d98-de07-474d-b7d4-7b98909dbf26 2026-02-22T22:04:38.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:04:38.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:04:38.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:04:38.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:04:38.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:04:38.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:04:38.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:04:38.748 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:04:38.749 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:04:38.749 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:04:38.749 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:04:38.750 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:04:38.750 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:04:38.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:04:38.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:04:38.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:04:38.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:04:38.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:04:38.849 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:04:39.059 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:04:39.059 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:04:39.059 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:04:39.059 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:04:39.059 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:04:39.060 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:39.060 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:04:39.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803793 2026-02-22T22:04:39.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803793 2026-02-22T22:04:39.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793' 2026-02-22T22:04:39.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:39.134 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:04:39.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=210453397507 2026-02-22T22:04:39.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 210453397507 2026-02-22T22:04:39.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-210453397507' 2026-02-22T22:04:39.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:39.209 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:04:39.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411342 2026-02-22T22:04:39.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411342 2026-02-22T22:04:39.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-210453397507 2-77309411342' 2026-02-22T22:04:39.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:39.284 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:04:39.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=171798691847 2026-02-22T22:04:39.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 171798691847 2026-02-22T22:04:39.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-210453397507 2-77309411342 3-171798691847' 2026-02-22T22:04:39.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:39.361 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803793 2026-02-22T22:04:39.361 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:39.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:04:39.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803793 2026-02-22T22:04:39.363 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:39.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803793 2026-02-22T22:04:39.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803793' 2026-02-22T22:04:39.365 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803793 2026-02-22T22:04:39.365 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:04:39.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803791 -lt 25769803793 2026-02-22T22:04:39.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:04:40.572 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:04:40.572 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:04:40.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803794 -lt 25769803793 2026-02-22T22:04:40.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:40.771 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-210453397507 2026-02-22T22:04:40.771 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:40.772 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:04:40.773 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-210453397507 2026-02-22T22:04:40.773 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:40.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=210453397507 2026-02-22T22:04:40.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 210453397507' 2026-02-22T22:04:40.775 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 210453397507 2026-02-22T22:04:40.775 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:04:40.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 210453397507 -lt 210453397507 2026-02-22T22:04:40.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:40.977 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411342 2026-02-22T22:04:40.978 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:40.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:04:40.979 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411342 2026-02-22T22:04:40.979 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:40.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411342 2026-02-22T22:04:40.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411342' 2026-02-22T22:04:40.981 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 77309411342 2026-02-22T22:04:40.981 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:04:41.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411342 -lt 77309411342 2026-02-22T22:04:41.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:41.188 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-171798691847 2026-02-22T22:04:41.188 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:41.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:04:41.189 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-171798691847 2026-02-22T22:04:41.190 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:41.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=171798691847 2026-02-22T22:04:41.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 171798691847' 2026-02-22T22:04:41.191 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 171798691847 2026-02-22T22:04:41.191 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:04:41.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 171798691847 -lt 171798691847 2026-02-22T22:04:41.394 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:04:41.394 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:04:41.395 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:04:41.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:04:41.650 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:04:41.651 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:04:41.651 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:04:41.651 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:04:41.651 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:04:41.651 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:04:41.651 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:04:41.840 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:04:41.840 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:04:41.841 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:04:41.841 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:04:42.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:04:42.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:04:42.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:04:42.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-22T22:04:42.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2026-02-22T22:04:42.363 INFO:tasks.workunit.client.0.trial034.stderr:noout is unset 2026-02-22T22:04:42.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-71671-0-10 fail 2026-02-22T22:04:42.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:04:42.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:04:42.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-71671-0-10 2026-02-22T22:04:42.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-22T22:04:42.375 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-22T22:04:42.375 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-71671-0-10 td/test-erasure-eio/COPY 2026-02-22T22:04:42.398 INFO:tasks.workunit.client.0.trial034.stderr:error getting pool-jerasure/obj-size-71671-0-10: (5) Input/output error 2026-02-22T22:04:42.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-22T22:04:42.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-22T22:04:42.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-71671-0-0 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-71671-0-0 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-71671-0-0 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:04:42.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-71671-0-0 td/test-erasure-eio/ORIGINAL 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-71671-0-0 td/test-erasure-eio 0 0 set 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-71671-0-0 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-71671-0-0 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:04:42.423 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-71671-0-0 2026-02-22T22:04:42.424 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-71671-0-0 2026-02-22T22:04:42.424 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:04:42.623 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:04:42.623 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:04:42.624 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:04:42.624 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:04:42.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2026-02-22T22:04:42.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2026-02-22T22:04:42.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2026-02-22T22:04:42.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2026-02-22T22:04:42.870 INFO:tasks.workunit.client.0.trial034.stderr:noout is set 2026-02-22T22:04:42.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2026-02-22T22:04:42.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2026-02-22T22:04:42.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2026-02-22T22:04:42.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-71671-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:42.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:04:42.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:04:42.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2026-02-22T22:04:42.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:04:42.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-71671-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:42.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:04:42.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:04:42.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2026-02-22T22:04:42.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:04:42.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-22T22:04:42.883 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:04:42.883 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:04:42.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:04:42.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:04:42.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:04:44.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:04:44.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-71671-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:44.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:04:44.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:04:44.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2026-02-22T22:04:44.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:04:44.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2026-02-22T22:04:44.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-71671-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:04:45.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:04:45.389 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:04:45.389 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:04:45.389 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:04:45.389 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:04:45.389 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:04:45.389 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:04:45.389 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:04:45.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:04:45.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:04:45.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:04:45.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:04:45.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:04:45.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:04:45.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:04:45.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:04:45.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:04:45.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:04:45.390 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:04:45.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2026-02-22T22:04:45.391 INFO:tasks.workunit.client.0.trial034.stderr:start osd.3 2026-02-22T22:04:45.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:04:45.392 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-22T22:04:45.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2026-02-22T22:04:45.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:04:45.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:04:45.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:04:45.410 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:45.406+0000 7f77053a38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:45.411 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:45.410+0000 7f77053a38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:45.412 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:45.410+0000 7f77053a38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:45.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2026-02-22T22:04:45.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:04:45.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:04:45.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:04:45.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:04:45.590 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:45.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:04:45.591 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:04:45.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:45.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:45.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:46.656 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:46.654+0000 7f77053a38c0 -1 Falling back to public interface 2026-02-22T22:04:46.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:46.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:46.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:04:46.791 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:04:46.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:46.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:46.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:47.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:47.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:47.993 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:04:47.994 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:04:47.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:47.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:48.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:49.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:49.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:49.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:04:49.192 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:04:49.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:49.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:49.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:49.658 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:49.654+0000 7f77053a38c0 -1 osd.3 54 log_to_monitors true 2026-02-22T22:04:50.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:50.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:50.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:04:50.394 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:04:50.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:50.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:04:50.587 INFO:tasks.workunit.client.0.trial034.stderr:osd.3 up in weight 1 up_from 59 up_thru 59 down_at 56 last_clean_interval [40,54) [v2:127.0.0.1:6826/3324108779,v1:127.0.0.1:6827/3324108779] [v2:127.0.0.1:6828/3324108779,v1:127.0.0.1:6829/3324108779] exists,up 7cfc75d6-c7bf-4610-ab5b-211726921771 2026-02-22T22:04:50.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:04:50.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:04:50.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:04:50.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:04:50.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:04:50.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:04:50.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:04:50.587 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:04:50.588 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:04:50.588 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:04:50.588 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:04:50.588 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:04:50.588 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:04:50.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:04:50.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:04:50.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:04:50.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:04:50.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:04:50.691 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:04:50.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:04:50.888 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:04:50.889 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:04:50.889 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:04:50.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:04:50.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:50.889 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:04:50.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803798 2026-02-22T22:04:50.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803798 2026-02-22T22:04:50.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803798' 2026-02-22T22:04:50.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:50.965 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:04:51.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=210453397511 2026-02-22T22:04:51.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 210453397511 2026-02-22T22:04:51.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803798 1-210453397511' 2026-02-22T22:04:51.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:51.040 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:04:51.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411346 2026-02-22T22:04:51.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411346 2026-02-22T22:04:51.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803798 1-210453397511 2-77309411346' 2026-02-22T22:04:51.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:04:51.115 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:04:51.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=253403070467 2026-02-22T22:04:51.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 253403070467 2026-02-22T22:04:51.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803798 1-210453397511 2-77309411346 3-253403070467' 2026-02-22T22:04:51.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:51.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803798 2026-02-22T22:04:51.193 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:51.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:04:51.195 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803798 2026-02-22T22:04:51.195 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:51.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803798 2026-02-22T22:04:51.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803798' 2026-02-22T22:04:51.196 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803798 2026-02-22T22:04:51.197 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:04:51.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803796 -lt 25769803798 2026-02-22T22:04:51.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:04:52.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:04:52.403 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:04:52.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803798 -lt 25769803798 2026-02-22T22:04:52.600 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:52.600 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-210453397511 2026-02-22T22:04:52.600 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:52.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:04:52.603 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-210453397511 2026-02-22T22:04:52.603 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:52.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=210453397511 2026-02-22T22:04:52.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 210453397511' 2026-02-22T22:04:52.605 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 210453397511 2026-02-22T22:04:52.606 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:04:52.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 210453397511 -lt 210453397511 2026-02-22T22:04:52.806 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:52.807 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411346 2026-02-22T22:04:52.807 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:52.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:04:52.809 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411346 2026-02-22T22:04:52.809 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:52.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411346 2026-02-22T22:04:52.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411346' 2026-02-22T22:04:52.810 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 77309411346 2026-02-22T22:04:52.811 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:04:53.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411346 -lt 77309411346 2026-02-22T22:04:53.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:04:53.022 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-253403070467 2026-02-22T22:04:53.023 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:04:53.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:04:53.024 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-253403070467 2026-02-22T22:04:53.024 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:04:53.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=253403070467 2026-02-22T22:04:53.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 253403070467' 2026-02-22T22:04:53.026 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 253403070467 2026-02-22T22:04:53.026 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:04:53.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 253403070467 -lt 253403070467 2026-02-22T22:04:53.219 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:04:53.220 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:04:53.220 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:04:53.484 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:04:53.484 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:04:53.485 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:04:53.485 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:04:53.485 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:04:53.485 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:04:53.485 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:04:53.485 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:04:53.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:04:53.674 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:04:53.674 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:04:53.674 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:04:53.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:04:53.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:04:53.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:04:53.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-22T22:04:53.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2026-02-22T22:04:54.195 INFO:tasks.workunit.client.0.trial034.stderr:noout is unset 2026-02-22T22:04:54.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-71671-0-0 2026-02-22T22:04:54.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:04:54.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:04:54.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-71671-0-0 2026-02-22T22:04:54.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:04:54.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:04:54.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-71671-0-0 td/test-erasure-eio/COPY 2026-02-22T22:04:54.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:04:54.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:04:54.230 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2026-02-22T22:04:54.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2026-02-22T22:04:54.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-71671-0-0 td/test-erasure-eio 1 0 set 2026-02-22T22:04:54.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-71671-0-0 2026-02-22T22:04:54.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2026-02-22T22:04:54.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2026-02-22T22:04:54.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2026-02-22T22:04:54.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2026-02-22T22:04:54.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2026-02-22T22:04:54.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2026-02-22T22:04:54.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2026-02-22T22:04:54.233 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2026-02-22T22:04:54.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2026-02-22T22:04:54.234 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-71671-0-0 2026-02-22T22:04:54.234 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:04:54.234 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-71671-0-0 2026-02-22T22:04:54.234 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-71671-0-0 2026-02-22T22:04:54.234 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:04:54.431 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:04:54.432 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:04:54.432 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:04:54.432 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:04:54.432 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2026-02-22T22:04:54.432 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2026-02-22T22:04:54.432 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2026-02-22T22:04:54.432 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2026-02-22T22:04:54.700 INFO:tasks.workunit.client.0.trial034.stderr:noout is set 2026-02-22T22:04:54.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2026-02-22T22:04:54.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2026-02-22T22:04:54.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2026-02-22T22:04:54.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-71671-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:54.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:04:54.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:04:54.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2026-02-22T22:04:54.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:04:54.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-71671-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:54.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:04:54.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:04:54.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2026-02-22T22:04:54.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:04:54.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-22T22:04:54.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:04:54.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:04:54.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:04:54.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:04:54.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:04:56.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:04:56.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-71671-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:56.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:04:56.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:04:56.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2026-02-22T22:04:56.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:04:56.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-22T22:04:56.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-71671-0-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:04:57.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-22T22:04:57.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:04:57.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:04:57.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2026-02-22T22:04:57.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:04:57.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:04:57.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:04:57.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:04:57.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:04:57.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:04:57.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:04:57.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:04:57.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:04:57.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:04:57.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:04:57.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:04:57.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:04:57.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:04:57.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:04:57.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:04:57.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:04:57.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2026-02-22T22:04:57.199 INFO:tasks.workunit.client.0.trial034.stderr:start osd.1 2026-02-22T22:04:57.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:04:57.200 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-22T22:04:57.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2026-02-22T22:04:57.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:04:57.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:04:57.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:04:57.219 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:57.218+0000 7f870e50f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:57.220 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:57.218+0000 7f870e50f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:57.222 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:57.218+0000 7f870e50f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:04:57.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2026-02-22T22:04:57.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:04:57.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:04:57.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:04:57.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:04:57.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:57.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:04:57.395 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:04:57.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:57.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:04:57.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:58.476 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:04:58.474+0000 7f870e50f8c0 -1 Falling back to public interface 2026-02-22T22:04:58.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:58.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:58.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:04:58.593 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:04:58.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:58.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:04:58.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:04:59.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:04:59.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:04:59.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:04:59.788 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:04:59.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:04:59.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:04:59.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:00.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:00.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:00.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:05:00.983 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:05:00.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:00.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:05:01.179 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:01.502 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:01.498+0000 7f870e50f8c0 -1 osd.1 62 log_to_monitors true 2026-02-22T22:05:02.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:02.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:02.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:05:02.181 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:05:02.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:02.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:05:02.375 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [49,62) [v2:127.0.0.1:6810/3309668947,v1:127.0.0.1:6811/3309668947] [v2:127.0.0.1:6812/3309668947,v1:127.0.0.1:6813/3309668947] exists,up 52a45d98-de07-474d-b7d4-7b98909dbf26 2026-02-22T22:05:02.375 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:05:02.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:05:02.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:05:02.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:05:02.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:05:02.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:05:02.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:05:02.376 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:05:02.377 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:05:02.377 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:05:02.377 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:05:02.377 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:05:02.378 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:05:02.473 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:05:02.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:05:02.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:05:02.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:05:02.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:05:02.474 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:05:02.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:05:02.662 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:05:02.663 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:05:02.663 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:05:02.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:05:02.663 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:02.663 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:05:02.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803802 2026-02-22T22:05:02.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803802 2026-02-22T22:05:02.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803802' 2026-02-22T22:05:02.743 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:02.743 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:05:02.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=292057776131 2026-02-22T22:05:02.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 292057776131 2026-02-22T22:05:02.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803802 1-292057776131' 2026-02-22T22:05:02.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:02.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:05:02.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411351 2026-02-22T22:05:02.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411351 2026-02-22T22:05:02.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803802 1-292057776131 2-77309411351' 2026-02-22T22:05:02.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:02.896 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:05:02.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=253403070471 2026-02-22T22:05:02.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 253403070471 2026-02-22T22:05:02.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803802 1-292057776131 2-77309411351 3-253403070471' 2026-02-22T22:05:02.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:02.973 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803802 2026-02-22T22:05:02.973 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:02.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:05:02.974 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803802 2026-02-22T22:05:02.975 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:02.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803802 2026-02-22T22:05:02.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803802' 2026-02-22T22:05:02.976 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803802 2026-02-22T22:05:02.977 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:05:03.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803800 -lt 25769803802 2026-02-22T22:05:03.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:05:04.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:05:04.178 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:05:04.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803800 -lt 25769803802 2026-02-22T22:05:04.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:05:05.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:05:05.380 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:05:05.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803803 -lt 25769803802 2026-02-22T22:05:05.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:05.576 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-292057776131 2026-02-22T22:05:05.576 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:05.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:05:05.578 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-292057776131 2026-02-22T22:05:05.578 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:05.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=292057776131 2026-02-22T22:05:05.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 292057776131' 2026-02-22T22:05:05.580 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 292057776131 2026-02-22T22:05:05.580 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:05:05.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 292057776131 -lt 292057776131 2026-02-22T22:05:05.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:05.786 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411351 2026-02-22T22:05:05.786 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:05.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:05:05.788 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411351 2026-02-22T22:05:05.788 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:05.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411351 2026-02-22T22:05:05.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411351' 2026-02-22T22:05:05.789 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 77309411351 2026-02-22T22:05:05.790 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:05:06.002 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411351 -lt 77309411351 2026-02-22T22:05:06.002 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:06.002 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-253403070471 2026-02-22T22:05:06.002 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:06.004 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:05:06.004 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-253403070471 2026-02-22T22:05:06.004 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:06.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=253403070471 2026-02-22T22:05:06.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 253403070471' 2026-02-22T22:05:06.006 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 253403070471 2026-02-22T22:05:06.006 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:05:06.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 253403070471 -lt 253403070471 2026-02-22T22:05:06.207 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:05:06.207 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:05:06.207 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:05:06.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:05:06.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:05:06.469 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:05:06.469 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:05:06.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:05:06.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:05:06.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:05:06.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:05:06.666 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:05:06.667 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:05:06.668 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:05:06.668 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:05:06.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:05:06.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:05:06.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:05:06.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-22T22:05:06.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2026-02-22T22:05:07.194 INFO:tasks.workunit.client.0.trial034.stderr:noout is unset 2026-02-22T22:05:07.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-71671-0-0 fail 2026-02-22T22:05:07.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:05:07.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:05:07.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-71671-0-0 2026-02-22T22:05:07.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-22T22:05:07.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-22T22:05:07.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-71671-0-0 td/test-erasure-eio/COPY 2026-02-22T22:05:07.222 INFO:tasks.workunit.client.0.trial034.stderr:error getting pool-jerasure/obj-size-71671-0-0: (5) Input/output error 2026-02-22T22:05:07.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-22T22:05:07.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-22T22:05:07.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:311: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2026-02-22T22:05:07.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-22T22:05:07.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2026-02-22T22:05:07.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2026-02-22T22:05:07.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2026-02-22T22:05:07.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2026-02-22T22:05:07.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2026-02-22T22:05:07.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2026-02-22T22:05:07.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-22T22:05:07.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-71671-0-256 2026-02-22T22:05:07.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-71671-0-256 2026-02-22T22:05:07.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:05:07.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:05:07.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-71671-0-256 2026-02-22T22:05:07.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:05:07.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:05:07.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:05:07.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:05:07.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:05:07.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:05:07.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:05:07.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:05:07.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-71671-0-256 td/test-erasure-eio/ORIGINAL 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-71671-0-256 td/test-erasure-eio 0 256 add 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-71671-0-256 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-71671-0-256 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-71671-0-256 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-71671-0-256 2026-02-22T22:05:07.247 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:05:07.442 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:05:07.442 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:05:07.442 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:05:07.442 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:05:07.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2026-02-22T22:05:07.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2026-02-22T22:05:07.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2026-02-22T22:05:07.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2026-02-22T22:05:07.701 INFO:tasks.workunit.client.0.trial034.stderr:noout is set 2026-02-22T22:05:07.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2026-02-22T22:05:07.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-71671-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-71671-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:05:07.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:05:09.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:05:09.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-71671-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:09.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:05:09.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:05:09.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2026-02-22T22:05:09.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:05:09.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2026-02-22T22:05:09.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-71671-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:09.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2026-02-22T22:05:09.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:05:09.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:05:09.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2026-02-22T22:05:09.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:05:09.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:05:09.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:05:09.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:05:09.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:05:09.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:05:09.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:05:09.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:05:09.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:05:09.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:05:09.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:05:09.874 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:05:09.874 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:05:09.874 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:05:09.874 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:05:09.875 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:09.875 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:09.877 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:05:09.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:05:09.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:05:09.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:05:09.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:05:09.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:05:09.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:05:09.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:05:09.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:05:09.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:05:09.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:05:09.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:05:09.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2026-02-22T22:05:09.878 INFO:tasks.workunit.client.0.trial034.stderr:start osd.3 2026-02-22T22:05:09.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:05:09.878 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-22T22:05:09.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2026-02-22T22:05:09.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:05:09.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:05:09.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:05:09.893 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:09.890+0000 7f7124d208c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:09.894 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:09.894+0000 7f7124d208c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:09.895 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:09.894+0000 7f7124d208c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:10.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2026-02-22T22:05:10.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:05:10.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:05:10.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:05:10.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:05:10.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:10.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:05:10.077 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:05:10.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:10.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:05:10.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:11.136 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:11.134+0000 7f7124d208c0 -1 Falling back to public interface 2026-02-22T22:05:11.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:11.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:11.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:05:11.273 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:05:11.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:11.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:05:11.479 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:12.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:12.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:12.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:05:12.481 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:05:12.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:12.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:05:12.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:13.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:13.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:13.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:05:13.678 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:05:13.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:13.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:05:13.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:14.167 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:14.166+0000 7f7124d208c0 -1 osd.3 71 log_to_monitors true 2026-02-22T22:05:14.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:14.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:14.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:05:14.871 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:05:14.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:14.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:05:15.064 INFO:tasks.workunit.client.0.trial034.stderr:osd.3 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [59,71) [v2:127.0.0.1:6826/282586976,v1:127.0.0.1:6827/282586976] [v2:127.0.0.1:6828/282586976,v1:127.0.0.1:6829/282586976] exists,up 7cfc75d6-c7bf-4610-ab5b-211726921771 2026-02-22T22:05:15.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:05:15.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:05:15.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:05:15.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:05:15.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:05:15.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:05:15.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:05:15.065 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:05:15.066 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:05:15.066 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:05:15.067 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:05:15.067 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:05:15.067 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:05:15.167 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:05:15.167 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:05:15.167 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:05:15.167 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:05:15.167 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:05:15.167 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:05:15.372 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:05:15.372 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:05:15.372 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:05:15.372 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:05:15.372 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:05:15.372 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:15.372 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:05:15.447 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803807 2026-02-22T22:05:15.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803807 2026-02-22T22:05:15.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803807' 2026-02-22T22:05:15.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:15.448 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:05:15.526 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=292057776135 2026-02-22T22:05:15.526 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 292057776135 2026-02-22T22:05:15.526 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803807 1-292057776135' 2026-02-22T22:05:15.526 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:15.526 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:05:15.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411355 2026-02-22T22:05:15.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411355 2026-02-22T22:05:15.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803807 1-292057776135 2-77309411355' 2026-02-22T22:05:15.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:15.602 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:05:15.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=330712481795 2026-02-22T22:05:15.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 330712481795 2026-02-22T22:05:15.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803807 1-292057776135 2-77309411355 3-330712481795' 2026-02-22T22:05:15.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:15.677 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803807 2026-02-22T22:05:15.677 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:15.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:05:15.678 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803807 2026-02-22T22:05:15.678 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:15.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803807 2026-02-22T22:05:15.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2026-02-22T22:05:15.679 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803807 2026-02-22T22:05:15.680 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:05:15.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803805 -lt 25769803807 2026-02-22T22:05:15.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:05:16.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:05:16.889 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:05:17.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803807 -lt 25769803807 2026-02-22T22:05:17.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:17.087 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-292057776135 2026-02-22T22:05:17.087 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:17.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:05:17.089 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-292057776135 2026-02-22T22:05:17.089 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:17.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=292057776135 2026-02-22T22:05:17.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 292057776135' 2026-02-22T22:05:17.090 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 292057776135 2026-02-22T22:05:17.091 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:05:17.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 292057776135 -lt 292057776135 2026-02-22T22:05:17.292 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:17.293 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411355 2026-02-22T22:05:17.293 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:17.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:05:17.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411355 2026-02-22T22:05:17.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:17.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411355 2026-02-22T22:05:17.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411355' 2026-02-22T22:05:17.296 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 77309411355 2026-02-22T22:05:17.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:05:17.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411355 -lt 77309411355 2026-02-22T22:05:17.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:17.492 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-330712481795 2026-02-22T22:05:17.492 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:17.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:05:17.494 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-330712481795 2026-02-22T22:05:17.494 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:17.495 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=330712481795 2026-02-22T22:05:17.495 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 330712481795' 2026-02-22T22:05:17.495 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 330712481795 2026-02-22T22:05:17.496 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:05:17.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 330712481795 -lt 330712481795 2026-02-22T22:05:17.685 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:05:17.685 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:05:17.685 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:05:17.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:05:17.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:05:17.943 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:05:17.943 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:05:17.943 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:05:17.943 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:05:17.944 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:05:17.944 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:05:18.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:05:18.136 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:05:18.137 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:05:18.137 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:05:18.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:05:18.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:05:18.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:05:18.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2026-02-22T22:05:18.397 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records in 2026-02-22T22:05:18.397 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records out 2026-02-22T22:05:18.397 INFO:tasks.workunit.client.0.trial034.stderr:256 bytes copied, 2.3945e-05 s, 10.7 MB/s 2026-02-22T22:05:18.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-71671-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:18.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:05:18.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:05:18.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2026-02-22T22:05:18.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:05:18.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-71671-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:18.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:05:18.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:05:18.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2026-02-22T22:05:18.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:05:18.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2026-02-22T22:05:18.398 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:05:18.398 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:05:18.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:05:18.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:05:18.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:05:19.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:05:19.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-71671-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:19.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:05:19.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:05:19.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2026-02-22T22:05:19.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:05:19.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2026-02-22T22:05:19.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-71671-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:20.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2026-02-22T22:05:20.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:05:20.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:05:20.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2026-02-22T22:05:20.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:05:20.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:05:20.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:05:20.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:05:20.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:05:20.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:05:20.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:05:20.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:05:20.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:05:20.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:05:20.909 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:05:20.909 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:05:20.909 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:05:20.909 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:05:20.909 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:05:20.910 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:20.910 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:20.910 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:05:20.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:05:20.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:05:20.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:05:20.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:05:20.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:05:20.910 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:05:20.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:05:20.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:05:20.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:05:20.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:05:20.911 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:05:20.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2026-02-22T22:05:20.913 INFO:tasks.workunit.client.0.trial034.stderr:start osd.3 2026-02-22T22:05:20.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:05:20.913 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2026-02-22T22:05:20.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2026-02-22T22:05:20.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:05:20.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:05:20.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:05:20.930 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:20.929+0000 7fd0fc5128c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:20.932 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:20.929+0000 7fd0fc5128c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:20.933 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:20.929+0000 7fd0fc5128c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:21.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2026-02-22T22:05:21.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:05:21.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:05:21.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:05:21.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:05:21.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:21.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:05:21.110 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:05:21.110 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:21.110 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:05:21.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:22.180 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:22.177+0000 7fd0fc5128c0 -1 Falling back to public interface 2026-02-22T22:05:22.304 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:22.304 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:22.304 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:05:22.304 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:05:22.304 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:22.304 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:05:22.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:23.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:23.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:23.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:05:23.504 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:05:23.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:23.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:05:23.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:24.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:24.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:24.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:05:24.700 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:05:24.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:24.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:05:24.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:25.221 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:25.217+0000 7fd0fc5128c0 -1 osd.3 79 log_to_monitors true 2026-02-22T22:05:25.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:25.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:25.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:05:25.895 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:05:25.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:25.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:05:26.088 INFO:tasks.workunit.client.0.trial034.stderr:osd.3 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [77,79) [v2:127.0.0.1:6826/3166139227,v1:127.0.0.1:6827/3166139227] [v2:127.0.0.1:6828/3166139227,v1:127.0.0.1:6829/3166139227] exists,up 7cfc75d6-c7bf-4610-ab5b-211726921771 2026-02-22T22:05:26.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:05:26.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:05:26.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:05:26.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:05:26.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:05:26.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:05:26.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:05:26.089 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:05:26.090 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:05:26.090 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:05:26.090 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:05:26.090 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:05:26.090 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:05:26.187 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:05:26.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:05:26.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:05:26.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:05:26.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:05:26.188 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:05:26.384 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:05:26.385 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:05:26.385 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:05:26.385 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:05:26.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:05:26.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:26.385 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:05:26.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803811 2026-02-22T22:05:26.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803811 2026-02-22T22:05:26.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803811' 2026-02-22T22:05:26.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:26.460 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:05:26.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=292057776139 2026-02-22T22:05:26.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 292057776139 2026-02-22T22:05:26.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803811 1-292057776139' 2026-02-22T22:05:26.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:26.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:05:26.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411359 2026-02-22T22:05:26.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411359 2026-02-22T22:05:26.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803811 1-292057776139 2-77309411359' 2026-02-22T22:05:26.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:26.611 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:05:26.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=356482285571 2026-02-22T22:05:26.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 356482285571 2026-02-22T22:05:26.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803811 1-292057776139 2-77309411359 3-356482285571' 2026-02-22T22:05:26.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:26.685 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803811 2026-02-22T22:05:26.686 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:26.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:05:26.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803811 2026-02-22T22:05:26.688 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:26.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803811 2026-02-22T22:05:26.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2026-02-22T22:05:26.690 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803811 2026-02-22T22:05:26.690 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:05:26.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803809 -lt 25769803811 2026-02-22T22:05:26.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:05:27.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:05:27.902 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:05:28.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803809 -lt 25769803811 2026-02-22T22:05:28.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:05:29.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:05:29.100 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:05:29.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803811 -lt 25769803811 2026-02-22T22:05:29.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:29.301 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-292057776139 2026-02-22T22:05:29.302 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:29.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:05:29.304 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-292057776139 2026-02-22T22:05:29.304 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:29.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=292057776139 2026-02-22T22:05:29.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 292057776139' 2026-02-22T22:05:29.306 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 292057776139 2026-02-22T22:05:29.306 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:05:29.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 292057776140 -lt 292057776139 2026-02-22T22:05:29.511 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:29.512 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411359 2026-02-22T22:05:29.512 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:29.513 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:05:29.514 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411359 2026-02-22T22:05:29.514 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:29.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411359 2026-02-22T22:05:29.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411359' 2026-02-22T22:05:29.515 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 77309411359 2026-02-22T22:05:29.516 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:05:29.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411360 -lt 77309411359 2026-02-22T22:05:29.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:29.724 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-356482285571 2026-02-22T22:05:29.724 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:29.725 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:05:29.726 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-356482285571 2026-02-22T22:05:29.726 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:29.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=356482285571 2026-02-22T22:05:29.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 356482285571' 2026-02-22T22:05:29.728 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 356482285571 2026-02-22T22:05:29.728 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:05:29.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 356482285571 -lt 356482285571 2026-02-22T22:05:29.926 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:05:29.926 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:05:29.926 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:05:30.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:05:30.179 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:05:30.180 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:05:30.180 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:05:30.180 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:05:30.180 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:05:30.180 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:05:30.180 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:05:30.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:05:30.371 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:05:30.372 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:05:30.372 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:05:30.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:05:30.626 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:05:30.626 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:05:30.626 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-22T22:05:30.626 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2026-02-22T22:05:30.936 INFO:tasks.workunit.client.0.trial034.stderr:noout is unset 2026-02-22T22:05:30.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-71671-0-256 2026-02-22T22:05:30.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:05:30.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:05:30.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-71671-0-256 2026-02-22T22:05:30.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:05:30.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:05:30.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-71671-0-256 td/test-erasure-eio/COPY 2026-02-22T22:05:30.975 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:05:30.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:05:30.978 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2026-02-22T22:05:30.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2026-02-22T22:05:30.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-71671-0-256 td/test-erasure-eio 1 256 add 2026-02-22T22:05:30.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-71671-0-256 2026-02-22T22:05:30.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2026-02-22T22:05:30.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2026-02-22T22:05:30.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2026-02-22T22:05:30.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2026-02-22T22:05:30.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2026-02-22T22:05:30.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2026-02-22T22:05:30.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2026-02-22T22:05:30.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2026-02-22T22:05:30.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2026-02-22T22:05:30.981 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-71671-0-256 2026-02-22T22:05:30.981 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:05:30.981 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-71671-0-256 2026-02-22T22:05:30.981 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-71671-0-256 2026-02-22T22:05:30.981 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:05:31.188 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:05:31.188 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:05:31.188 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:05:31.188 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:05:31.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2026-02-22T22:05:31.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2026-02-22T22:05:31.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2026-02-22T22:05:31.188 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2026-02-22T22:05:31.444 INFO:tasks.workunit.client.0.trial034.stderr:noout is set 2026-02-22T22:05:31.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2026-02-22T22:05:31.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-71671-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:31.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:05:31.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:05:31.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2026-02-22T22:05:31.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:05:31.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-71671-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:31.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:05:31.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:05:31.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2026-02-22T22:05:31.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:05:31.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-22T22:05:31.458 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:05:31.458 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:05:31.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:05:31.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:05:31.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:05:32.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:05:32.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-71671-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:32.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:05:32.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:05:32.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2026-02-22T22:05:32.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:05:32.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-22T22:05:32.767 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-71671-0-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:33.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-22T22:05:33.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:05:33.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:05:33.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2026-02-22T22:05:33.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:05:33.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:05:33.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:05:33.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:05:33.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:05:33.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:05:33.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:05:33.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:05:33.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:05:33.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:05:33.632 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:05:33.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:05:33.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:05:33.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:05:33.635 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:05:33.635 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:33.635 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:33.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:05:33.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:05:33.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:05:33.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:05:33.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:05:33.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:05:33.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:05:33.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:05:33.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:05:33.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:05:33.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:05:33.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:05:33.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2026-02-22T22:05:33.636 INFO:tasks.workunit.client.0.trial034.stderr:start osd.1 2026-02-22T22:05:33.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:05:33.637 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-22T22:05:33.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2026-02-22T22:05:33.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:05:33.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:05:33.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:05:33.653 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:33.649+0000 7f0168ae78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:33.655 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:33.653+0000 7f0168ae78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:33.655 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:33.653+0000 7f0168ae78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:33.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2026-02-22T22:05:33.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:05:33.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:05:33.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:05:33.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:05:33.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:33.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:05:33.837 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:05:33.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:33.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:05:34.037 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:34.896 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:34.893+0000 7f0168ae78c0 -1 Falling back to public interface 2026-02-22T22:05:35.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:35.039 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:35.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:05:35.040 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:05:35.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:35.040 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:05:35.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:36.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:36.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:36.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:05:36.242 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:05:36.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:36.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:05:36.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:37.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:37.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:37.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:05:37.438 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:05:37.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:37.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:05:37.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:38.164 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:38.161+0000 7f0168ae78c0 -1 osd.1 86 log_to_monitors true 2026-02-22T22:05:38.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:38.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:38.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:05:38.642 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:05:38.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:38.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:05:38.841 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 up in weight 1 up_from 92 up_thru 92 down_at 89 last_clean_interval [68,86) [v2:127.0.0.1:6810/1385330673,v1:127.0.0.1:6811/1385330673] [v2:127.0.0.1:6812/1385330673,v1:127.0.0.1:6813/1385330673] exists,up 52a45d98-de07-474d-b7d4-7b98909dbf26 2026-02-22T22:05:38.841 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:05:38.841 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:05:38.841 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:05:38.841 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:05:38.841 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:05:38.841 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:05:38.841 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:05:38.841 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:05:38.842 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:05:38.842 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:05:38.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:05:38.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:05:38.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:05:38.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:05:38.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:05:38.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:05:38.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:05:38.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:05:38.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:05:39.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:05:39.148 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:05:39.148 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:05:39.149 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:05:39.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:05:39.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:39.149 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:05:39.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803815 2026-02-22T22:05:39.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803815 2026-02-22T22:05:39.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803815' 2026-02-22T22:05:39.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:39.225 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:05:39.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=395136991235 2026-02-22T22:05:39.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 395136991235 2026-02-22T22:05:39.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803815 1-395136991235' 2026-02-22T22:05:39.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:39.300 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:05:39.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411364 2026-02-22T22:05:39.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411364 2026-02-22T22:05:39.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803815 1-395136991235 2-77309411364' 2026-02-22T22:05:39.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:39.376 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:05:39.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=356482285575 2026-02-22T22:05:39.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 356482285575 2026-02-22T22:05:39.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803815 1-395136991235 2-77309411364 3-356482285575' 2026-02-22T22:05:39.453 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:39.454 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803815 2026-02-22T22:05:39.454 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:39.454 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:05:39.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803815 2026-02-22T22:05:39.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:39.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803815 2026-02-22T22:05:39.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803815' 2026-02-22T22:05:39.457 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803815 2026-02-22T22:05:39.457 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:05:39.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803813 -lt 25769803815 2026-02-22T22:05:39.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:05:40.653 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:05:40.653 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:05:40.857 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803816 -lt 25769803815 2026-02-22T22:05:40.857 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:40.858 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-395136991235 2026-02-22T22:05:40.858 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:40.859 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:05:40.860 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-395136991235 2026-02-22T22:05:40.860 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:40.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=395136991235 2026-02-22T22:05:40.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 395136991235' 2026-02-22T22:05:40.862 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 395136991235 2026-02-22T22:05:40.862 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:05:41.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 395136991235 -lt 395136991235 2026-02-22T22:05:41.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:41.070 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411364 2026-02-22T22:05:41.070 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:41.071 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:05:41.072 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411364 2026-02-22T22:05:41.072 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:41.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411364 2026-02-22T22:05:41.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411364' 2026-02-22T22:05:41.073 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 77309411364 2026-02-22T22:05:41.074 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:05:41.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411364 -lt 77309411364 2026-02-22T22:05:41.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:41.282 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-356482285575 2026-02-22T22:05:41.282 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:41.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:05:41.284 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-356482285575 2026-02-22T22:05:41.285 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:41.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=356482285575 2026-02-22T22:05:41.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 356482285575' 2026-02-22T22:05:41.286 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 356482285575 2026-02-22T22:05:41.286 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:05:41.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 356482285576 -lt 356482285575 2026-02-22T22:05:41.487 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:05:41.487 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:05:41.487 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:05:41.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:05:41.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:05:41.742 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:05:41.742 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:05:41.742 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:05:41.742 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:05:41.742 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:05:41.743 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:05:41.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:05:41.932 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:05:41.933 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:05:41.933 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:05:42.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:05:42.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:05:42.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:05:42.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2026-02-22T22:05:42.192 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records in 2026-02-22T22:05:42.192 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records out 2026-02-22T22:05:42.193 INFO:tasks.workunit.client.0.trial034.stderr:256 bytes copied, 2.3213e-05 s, 11.0 MB/s 2026-02-22T22:05:42.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-71671-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:42.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:05:42.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:05:42.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2026-02-22T22:05:42.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:05:42.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-71671-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:42.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:05:42.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:05:42.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2026-02-22T22:05:42.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:05:42.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-22T22:05:42.194 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:05:42.194 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:05:42.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:05:42.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:05:42.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:05:43.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:05:43.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-71671-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:43.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:05:43.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:05:43.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2026-02-22T22:05:43.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:05:43.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-22T22:05:43.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-71671-0-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:05:44.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-22T22:05:44.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:05:44.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:05:44.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:44.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:05:44.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:05:44.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:05:44.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:05:44.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:05:44.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:05:44.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:05:44.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:05:44.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:05:44.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:05:44.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:05:44.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:05:44.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2026-02-22T22:05:44.671 INFO:tasks.workunit.client.0.trial034.stderr:start osd.1 2026-02-22T22:05:44.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:05:44.671 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-22T22:05:44.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2026-02-22T22:05:44.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:05:44.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:05:44.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:05:44.693 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:44.689+0000 7fee02f378c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:44.695 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:44.693+0000 7fee02f378c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:44.696 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:44.693+0000 7fee02f378c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:44.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2026-02-22T22:05:44.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:05:44.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:05:44.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:05:44.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:05:44.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:44.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:05:44.871 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:05:44.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:44.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:05:45.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:45.960 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:45.957+0000 7fee02f378c0 -1 Falling back to public interface 2026-02-22T22:05:46.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:46.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:46.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:05:46.066 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:05:46.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:46.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:05:46.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:47.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:47.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:47.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:05:47.261 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:05:47.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:47.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:05:47.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:48.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:48.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:48.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:05:48.459 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:05:48.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:48.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:05:48.652 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:05:49.219 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:49.217+0000 7fee02f378c0 -1 osd.1 93 log_to_monitors true 2026-02-22T22:05:49.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:05:49.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:05:49.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:05:49.655 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:05:49.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:05:49.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:05:49.850 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 up in weight 1 up_from 97 up_thru 97 down_at 94 last_clean_interval [92,93) [v2:127.0.0.1:6810/287224283,v1:127.0.0.1:6811/287224283] [v2:127.0.0.1:6812/287224283,v1:127.0.0.1:6813/287224283] exists,up 52a45d98-de07-474d-b7d4-7b98909dbf26 2026-02-22T22:05:49.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:05:49.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:05:49.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:05:49.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:05:49.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:05:49.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:05:49.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:05:49.851 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:05:49.852 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:05:49.852 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:05:49.852 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:05:49.852 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:05:49.852 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:05:49.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:05:49.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:05:49.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:05:49.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:05:49.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:05:49.951 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:05:50.143 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:05:50.143 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:05:50.143 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:05:50.143 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:05:50.143 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:05:50.143 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:50.143 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:05:50.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803820 2026-02-22T22:05:50.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803820 2026-02-22T22:05:50.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803820' 2026-02-22T22:05:50.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:50.216 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:05:50.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=416611827715 2026-02-22T22:05:50.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 416611827715 2026-02-22T22:05:50.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803820 1-416611827715' 2026-02-22T22:05:50.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:50.290 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:05:50.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411368 2026-02-22T22:05:50.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411368 2026-02-22T22:05:50.365 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803820 1-416611827715 2-77309411368' 2026-02-22T22:05:50.365 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:05:50.365 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:05:50.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=356482285580 2026-02-22T22:05:50.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 356482285580 2026-02-22T22:05:50.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803820 1-416611827715 2-77309411368 3-356482285580' 2026-02-22T22:05:50.439 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:50.439 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803820 2026-02-22T22:05:50.439 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:50.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:05:50.440 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803820 2026-02-22T22:05:50.441 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:50.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803820 2026-02-22T22:05:50.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803820' 2026-02-22T22:05:50.442 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803820 2026-02-22T22:05:50.443 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:05:50.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803820 -lt 25769803820 2026-02-22T22:05:50.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:50.639 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-416611827715 2026-02-22T22:05:50.639 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:50.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:05:50.641 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-416611827715 2026-02-22T22:05:50.641 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:50.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=416611827715 2026-02-22T22:05:50.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 416611827715' 2026-02-22T22:05:50.642 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 416611827715 2026-02-22T22:05:50.643 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:05:50.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 416611827715 -lt 416611827715 2026-02-22T22:05:50.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:50.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411368 2026-02-22T22:05:50.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:50.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:05:50.845 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411368 2026-02-22T22:05:50.845 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:50.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411368 2026-02-22T22:05:50.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411368' 2026-02-22T22:05:50.847 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 77309411368 2026-02-22T22:05:50.847 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:05:51.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411368 -lt 77309411368 2026-02-22T22:05:51.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:05:51.046 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-356482285580 2026-02-22T22:05:51.046 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:05:51.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:05:51.048 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-356482285580 2026-02-22T22:05:51.048 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:05:51.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=356482285580 2026-02-22T22:05:51.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 356482285580' 2026-02-22T22:05:51.049 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 356482285580 2026-02-22T22:05:51.049 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:05:51.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 356482285579 -lt 356482285580 2026-02-22T22:05:51.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:05:52.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:05:52.247 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:05:52.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 356482285579 -lt 356482285580 2026-02-22T22:05:52.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:05:53.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:05:53.446 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:05:53.647 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 356482285580 -lt 356482285580 2026-02-22T22:05:53.648 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:05:53.648 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:05:53.648 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:05:53.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:05:53.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:05:53.905 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:05:53.905 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:05:53.905 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:05:53.905 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:05:53.905 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:05:53.905 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:05:54.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:05:54.098 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:05:54.099 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:05:54.099 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:05:54.354 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:05:54.354 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:05:54.354 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:05:54.354 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-22T22:05:54.354 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2026-02-22T22:05:54.647 INFO:tasks.workunit.client.0.trial034.stderr:noout is unset 2026-02-22T22:05:54.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-71671-0-256 fail 2026-02-22T22:05:54.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:05:54.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:05:54.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-71671-0-256 2026-02-22T22:05:54.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-22T22:05:54.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-22T22:05:54.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-71671-0-256 td/test-erasure-eio/COPY 2026-02-22T22:05:54.681 INFO:tasks.workunit.client.0.trial034.stderr:error getting pool-jerasure/obj-size-71671-0-256: (5) Input/output error 2026-02-22T22:05:54.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-22T22:05:54.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-22T22:05:54.684 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:312: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2026-02-22T22:05:54.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:05:54.685 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T22:05:54.898 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T22:05:54.906 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T22:05:55.163 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T22:05:55.174 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T22:05:55.174 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:05:55.174 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:05:55.174 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:05:55.175 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:05:55.175 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:05:55.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:05:55.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:05:55.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:05:55.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:05:55.486 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:05:55.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:05:55.487 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:05:55.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:05:55.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:05:55.490 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:05:55.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:05:55.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:05:55.492 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:05:55.492 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:05:55.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:05:55.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:05:55.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:05:55.517 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:05:55.517 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:55.518 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:55.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:05:55.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:05:55.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:05:55.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T22:05:55.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T22:05:55.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T22:05:55.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T22:05:55.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:05:55.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:05:55.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:05:55.521 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:05:55.521 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:05:55.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:05:55.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:05:55.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:05:55.522 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:05:55.523 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:05:55.523 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:05:55.524 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:05:55.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:05:55.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:05:55.525 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:05:55.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:05:55.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:05:55.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:05:55.528 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:05:55.529 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:05:55.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:05:55.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:05:55.531 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:05:55.531 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:55.531 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:55.531 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:05:55.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:05:55.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:05:55.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T22:05:55.534 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:05:55.534 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:55.534 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:55.534 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T22:05:55.535 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:05:55.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:05:55.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:05:55.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T22:05:55.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T22:05:55.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T22:05:55.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:05:55.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:05:55.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:05:55.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T22:05:55.537 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T22:05:55.558 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:05:55.558 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:05:55.558 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:05:55.559 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:05:55.559 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:55.559 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:55.559 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:05:55.559 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:05:55.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:05:55.590 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:05:55.591 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:05:55.591 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:05:55.591 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:05:55.591 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:05:55.592 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:05:55.592 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:05:55.592 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:05:55.592 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:05:55.592 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:55.593 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:55.593 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:05:55.593 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:05:55.593 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T22:05:55.649 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:05:55.650 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:05:55.650 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:05:55.650 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:05:55.650 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:05:55.651 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:05:55.651 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:05:55.652 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:05:55.652 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:05:55.652 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:55.652 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:55.652 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:05:55.652 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:05:55.652 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T22:05:55.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T22:05:55.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T22:05:55.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:05:55.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:05:55.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:05:55.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T22:05:55.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:05:55.817 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:05:55.817 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:05:55.817 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:05:55.818 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:05:55.818 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:55.818 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:55.818 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:05:55.818 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:05:55.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:05:55.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T22:05:55.838 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:05:55.941 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T22:05:55.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:05:56.951 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T22:05:56.951 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:05:56.952 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:05:56.952 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:05:56.952 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:56.952 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:56.952 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:05:56.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T22:05:56.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T22:05:57.002 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:05:57.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T22:05:57.010 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:05:55.577+0000 7f19131e0ec0 0 load: jerasure load: lrc 2026-02-22T22:05:57.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2026-02-22T22:05:57.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:316: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2026-02-22T22:05:57.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:317: TEST_rados_get_bad_size_shard_1: setup_osds 4 2026-02-22T22:05:57.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-22T22:05:57.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T22:05:57.011 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-22T22:05:57.012 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:05:57.014 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:05:57.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T22:05:57.016 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:05:57.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3ee9f565-4da4-4e51-ba0d-c3add359e46a 2026-02-22T22:05:57.017 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 3ee9f565-4da4-4e51-ba0d-c3add359e46a 2026-02-22T22:05:57.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3ee9f565-4da4-4e51-ba0d-c3add359e46a' 2026-02-22T22:05:57.018 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:05:57.028 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDFfZtpqsfIARAAbJ/JgKxVyuznT6SjTBc3Uw== 2026-02-22T22:05:57.028 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDFfZtpqsfIARAAbJ/JgKxVyuznT6SjTBc3Uw=="}' 2026-02-22T22:05:57.028 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3ee9f565-4da4-4e51-ba0d-c3add359e46a -i td/test-erasure-eio/0/new.json 2026-02-22T22:05:57.139 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:05:57.150 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T22:05:57.151 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFfZtpqsfIARAAbJ/JgKxVyuznT6SjTBc3Uw== --osd-uuid 3ee9f565-4da4-4e51-ba0d-c3add359e46a 2026-02-22T22:05:57.167 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:57.165+0000 7fda008038c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:57.169 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:57.165+0000 7fda008038c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:57.169 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:57.165+0000 7fda008038c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:05:57.170 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:57.165+0000 7fda008038c0 -1 bdev(0x563e4564dc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:05:57.170 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:05:57.169+0000 7fda008038c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T22:05:59.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T22:05:59.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:05:59.732 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:05:59.732 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T22:05:59.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:06:00.002 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T22:06:00.003 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:06:00.003 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:06:00.003 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:06:00.003 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:06:00.003 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:06:00.027 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:00.025+0000 7f3b4a98a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:00.029 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:00.025+0000 7f3b4a98a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:00.030 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:00.029+0000 7f3b4a98a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:00.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:06:00.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:06:00.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:06:00.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:06:00.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:06:00.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:00.199 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:06:00.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:06:00.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:00.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:06:00.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:01.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:01.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:01.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:06:01.397 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:06:01.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:01.397 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:06:01.591 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:01.768 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:01.765+0000 7f3b4a98a8c0 -1 Falling back to public interface 2026-02-22T22:06:02.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:02.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:02.593 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:06:02.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:06:02.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:02.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:06:02.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:03.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:03.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:03.789 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:06:03.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:06:03.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:03.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:06:03.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:04.763 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:04.761+0000 7f3b4a98a8c0 -1 osd.0 0 log_to_monitors true 2026-02-22T22:06:04.985 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:04.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:04.986 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:06:04.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:06:04.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:04.986 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:06:05.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:06.197 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:06.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:06.198 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:06:06.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:06:06.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:06.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:06:06.395 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/925506597,v1:127.0.0.1:6803/925506597] [v2:127.0.0.1:6804/925506597,v1:127.0.0.1:6805/925506597] exists,up 3ee9f565-4da4-4e51-ba0d-c3add359e46a 2026-02-22T22:06:06.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:06:06.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:06:06.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:06:06.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:06:06.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T22:06:06.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:06:06.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:06:06.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:06:06.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:06:06.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:06:06.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:06:06.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:06:06.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:06:06.400 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:06:06.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:06:06.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:06:06.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:06:06.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:06:06.401 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:06:06.401 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:06:06.401 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:06:06.401 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:06:06.401 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:06:06.401 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:06:06.401 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:06:06.401 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:06:06.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:06:06.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:06:06.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:06:06.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:06:06.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:06:06.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:06:06.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:06:06.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:06:06.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:06:06.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:06:06.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:06:06.402 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:06:06.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:06:06.403 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:06:06.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5b0429d8-014c-4797-98aa-6e2587f9225f 2026-02-22T22:06:06.403 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5b0429d8-014c-4797-98aa-6e2587f9225f' 2026-02-22T22:06:06.404 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 5b0429d8-014c-4797-98aa-6e2587f9225f 2026-02-22T22:06:06.404 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:06:06.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOfZtpDMmwGBAABxlt/YBhgJPp0MyXS525xQ== 2026-02-22T22:06:06.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOfZtpDMmwGBAABxlt/YBhgJPp0MyXS525xQ=="}' 2026-02-22T22:06:06.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5b0429d8-014c-4797-98aa-6e2587f9225f -i td/test-erasure-eio/1/new.json 2026-02-22T22:06:06.613 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:06:06.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T22:06:06.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOfZtpDMmwGBAABxlt/YBhgJPp0MyXS525xQ== --osd-uuid 5b0429d8-014c-4797-98aa-6e2587f9225f 2026-02-22T22:06:06.639 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:06.636+0000 7fc4c6e838c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:06.641 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:06.636+0000 7fc4c6e838c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:06.642 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:06.640+0000 7fc4c6e838c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:06.643 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:06.640+0000 7fc4c6e838c0 -1 bdev(0x560b0021dc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:06:06.643 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:06.640+0000 7fc4c6e838c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T22:06:08.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T22:06:08.939 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:06:08.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:06:08.940 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T22:06:08.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:06:09.220 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T22:06:09.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:06:09.221 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:06:09.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:06:09.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:06:09.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:06:09.236 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:09.232+0000 7fd8283e08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:09.237 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:09.232+0000 7fd8283e08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:09.238 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:09.236+0000 7fd8283e08c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:09.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:06:09.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:06:09.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:06:09.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:06:09.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:06:09.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:09.422 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:06:09.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:06:09.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:09.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:06:09.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:10.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:10.623 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:10.623 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:06:10.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:06:10.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:10.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:06:10.772 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:10.768+0000 7fd8283e08c0 -1 Falling back to public interface 2026-02-22T22:06:10.817 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:11.818 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:11.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:11.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:06:11.819 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:06:11.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:11.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:06:12.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:13.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:13.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:13.017 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:06:13.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:06:13.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:13.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:06:13.210 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:13.775 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:13.772+0000 7fd8283e08c0 -1 osd.1 0 log_to_monitors true 2026-02-22T22:06:14.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:14.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:14.212 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:06:14.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:06:14.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:14.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:06:14.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:15.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:15.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:15.407 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:06:15.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:06:15.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:15.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:06:15.599 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/673904777,v1:127.0.0.1:6811/673904777] [v2:127.0.0.1:6812/673904777,v1:127.0.0.1:6813/673904777] exists,up 5b0429d8-014c-4797-98aa-6e2587f9225f 2026-02-22T22:06:15.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:06:15.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:06:15.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:06:15.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:06:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T22:06:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:06:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:06:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:06:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:06:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:06:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:06:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:06:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:06:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:06:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:06:15.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:06:15.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:06:15.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:06:15.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:06:15.605 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:06:15.605 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:06:15.605 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:06:15.605 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:06:15.605 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:06:15.605 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:06:15.605 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:06:15.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:06:15.605 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:06:15.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:06:15.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:06:15.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:06:15.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:06:15.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:06:15.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:06:15.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:06:15.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:06:15.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:06:15.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:06:15.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:06:15.606 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:06:15.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fa1dd089-f822-4d84-b6b5-4516e45037e0 2026-02-22T22:06:15.607 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 fa1dd089-f822-4d84-b6b5-4516e45037e0 2026-02-22T22:06:15.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fa1dd089-f822-4d84-b6b5-4516e45037e0' 2026-02-22T22:06:15.608 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:06:15.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXfZtp+zLRJBAAS9WX2sQfCRt5rz2SkT6ebg== 2026-02-22T22:06:15.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXfZtp+zLRJBAAS9WX2sQfCRt5rz2SkT6ebg=="}' 2026-02-22T22:06:15.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fa1dd089-f822-4d84-b6b5-4516e45037e0 -i td/test-erasure-eio/2/new.json 2026-02-22T22:06:15.820 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:06:15.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T22:06:15.830 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXfZtp+zLRJBAAS9WX2sQfCRt5rz2SkT6ebg== --osd-uuid fa1dd089-f822-4d84-b6b5-4516e45037e0 2026-02-22T22:06:15.845 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:15.840+0000 7ffb87cce8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:15.846 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:15.844+0000 7ffb87cce8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:15.847 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:15.844+0000 7ffb87cce8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:15.848 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:15.844+0000 7ffb87cce8c0 -1 bdev(0x559bc03f3c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:06:15.848 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:15.844+0000 7ffb87cce8c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T22:06:18.127 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T22:06:18.127 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:06:18.128 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:06:18.129 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T22:06:18.129 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:06:18.384 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T22:06:18.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:06:18.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:06:18.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:06:18.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:06:18.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:06:18.406 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:18.400+0000 7fc11414e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:18.407 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:18.404+0000 7fc11414e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:18.408 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:18.404+0000 7fc11414e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:18.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:06:18.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:06:18.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:06:18.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:06:18.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:06:18.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:18.586 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:06:18.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:06:18.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:18.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:06:18.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:19.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:19.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:19.782 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:06:19.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:06:19.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:19.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:06:19.891 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:19.888+0000 7fc11414e8c0 -1 Falling back to public interface 2026-02-22T22:06:19.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:20.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:20.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:20.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:06:20.980 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:06:20.980 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:20.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:06:21.172 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:22.173 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:22.173 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:22.174 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:06:22.174 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:06:22.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:22.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:06:22.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:23.121 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:23.116+0000 7fc11414e8c0 -1 osd.2 0 log_to_monitors true 2026-02-22T22:06:23.368 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:23.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:23.369 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:06:23.369 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:06:23.370 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:23.370 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:06:23.569 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:24.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:24.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:24.570 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:06:24.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:06:24.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:24.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:06:24.770 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3161859131,v1:127.0.0.1:6819/3161859131] [v2:127.0.0.1:6820/3161859131,v1:127.0.0.1:6821/3161859131] exists,up fa1dd089-f822-4d84-b6b5-4516e45037e0 2026-02-22T22:06:24.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:06:24.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:06:24.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:06:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:06:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T22:06:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:06:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:06:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T22:06:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:06:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:06:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:06:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:06:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:06:24.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:06:24.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:06:24.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:06:24.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:06:24.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:06:24.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:06:24.776 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:06:24.776 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:06:24.776 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:06:24.776 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:06:24.776 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:06:24.777 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:06:24.777 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:06:24.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:06:24.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:06:24.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:06:24.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:06:24.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:06:24.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:06:24.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:06:24.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:06:24.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:06:24.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:06:24.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:06:24.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:06:24.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:06:24.779 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:06:24.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a0670a29-861d-45fa-bd3e-11039db157ef 2026-02-22T22:06:24.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 a0670a29-861d-45fa-bd3e-11039db157ef' 2026-02-22T22:06:24.780 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 a0670a29-861d-45fa-bd3e-11039db157ef 2026-02-22T22:06:24.780 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:06:24.788 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDgfZtpncoOLxAALDfUZAFtg6AqHGkLLOA6Ow== 2026-02-22T22:06:24.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDgfZtpncoOLxAALDfUZAFtg6AqHGkLLOA6Ow=="}' 2026-02-22T22:06:24.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a0670a29-861d-45fa-bd3e-11039db157ef -i td/test-erasure-eio/3/new.json 2026-02-22T22:06:24.995 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:06:25.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T22:06:25.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDgfZtpncoOLxAALDfUZAFtg6AqHGkLLOA6Ow== --osd-uuid a0670a29-861d-45fa-bd3e-11039db157ef 2026-02-22T22:06:25.031 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:25.028+0000 7f11beda78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:25.032 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:25.028+0000 7f11beda78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:25.034 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:25.028+0000 7f11beda78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:25.034 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:25.028+0000 7f11beda78c0 -1 bdev(0x55b9dbaafc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T22:06:25.034 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:25.028+0000 7f11beda78c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T22:06:27.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T22:06:27.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:06:27.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T22:06:27.325 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T22:06:27.326 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:06:27.594 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T22:06:27.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T22:06:27.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:06:27.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:06:27.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:06:27.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:06:27.610 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:27.604+0000 7f9ec24f38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:27.611 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:27.608+0000 7f9ec24f38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:27.612 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:27.608+0000 7f9ec24f38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:27.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T22:06:27.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:06:27.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:06:27.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:06:27.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:06:27.800 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:27.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:06:27.801 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:06:27.801 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:27.802 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:06:27.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:28.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:28.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:28.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:06:28.997 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:06:28.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:28.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:06:29.119 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:29.116+0000 7f9ec24f38c0 -1 Falling back to public interface 2026-02-22T22:06:29.191 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:30.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:30.193 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:30.193 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:06:30.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:06:30.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:30.194 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:06:30.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:31.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:31.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:31.388 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:06:31.389 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:06:31.389 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:31.389 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:06:31.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:32.333 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:32.328+0000 7f9ec24f38c0 -1 osd.3 0 log_to_monitors true 2026-02-22T22:06:32.579 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:32.579 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:32.579 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:06:32.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:06:32.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:32.580 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:06:32.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:33.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:33.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:33.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:06:33.781 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:06:33.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:33.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:06:33.974 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3375771899,v1:127.0.0.1:6827/3375771899] [v2:127.0.0.1:6828/3375771899,v1:127.0.0.1:6829/3375771899] exists,up a0670a29-861d-45fa-bd3e-11039db157ef 2026-02-22T22:06:33.975 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:06:33.975 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:06:33.975 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:06:33.976 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T22:06:33.976 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:06:33.976 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:06:33.976 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:06:33.976 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:06:33.976 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:06:33.977 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T22:06:33.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T22:06:33.977 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T22:06:34.037 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:06:34.043 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T22:06:34.045 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:06:01.769+0000 7f3b4a98a8c0 0 load: jerasure load: lrc 2026-02-22T22:06:34.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:319: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2026-02-22T22:06:34.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:320: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2026-02-22T22:06:34.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:06:34.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T22:06:34.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-22T22:06:34.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T22:06:34.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-22T22:06:34.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T22:06:34.045 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-22T22:06:34.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T22:06:34.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T22:06:34.645 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T22:06:34.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:06:35.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T22:06:35.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:06:35.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:06:35.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:06:35.659 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:06:35.660 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:06:35.660 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:06:35.660 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:06:35.661 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:06:35.661 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:06:35.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:06:35.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:06:35.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:06:35.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:06:35.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:06:35.769 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:06:35.970 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:06:35.970 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:06:35.970 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:06:35.970 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:06:35.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:06:35.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:06:35.971 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:06:36.043 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803785 2026-02-22T22:06:36.044 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803785 2026-02-22T22:06:36.044 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803785' 2026-02-22T22:06:36.044 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:06:36.044 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:06:36.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607559 2026-02-22T22:06:36.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607559 2026-02-22T22:06:36.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803785 1-51539607559' 2026-02-22T22:06:36.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:06:36.121 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:06:36.195 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=73014444037 2026-02-22T22:06:36.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 73014444037 2026-02-22T22:06:36.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803785 1-51539607559 2-73014444037' 2026-02-22T22:06:36.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:06:36.196 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:06:36.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182403 2026-02-22T22:06:36.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182403 2026-02-22T22:06:36.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803785 1-51539607559 2-73014444037 3-107374182403' 2026-02-22T22:06:36.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:06:36.273 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803785 2026-02-22T22:06:36.273 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:06:36.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:06:36.275 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803785 2026-02-22T22:06:36.275 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:06:36.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803785 2026-02-22T22:06:36.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2026-02-22T22:06:36.277 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803785 2026-02-22T22:06:36.277 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:06:36.479 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803785 -lt 25769803785 2026-02-22T22:06:36.479 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:06:36.480 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607559 2026-02-22T22:06:36.480 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:06:36.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:06:36.481 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607559 2026-02-22T22:06:36.482 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:06:36.483 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607559 2026-02-22T22:06:36.483 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607559 2026-02-22T22:06:36.484 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607559' 2026-02-22T22:06:36.484 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:06:36.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607557 -lt 51539607559 2026-02-22T22:06:36.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:06:37.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:06:37.696 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:06:37.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607557 -lt 51539607559 2026-02-22T22:06:37.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:06:38.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:06:38.898 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:06:39.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607559 -lt 51539607559 2026-02-22T22:06:39.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:06:39.091 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-73014444037 2026-02-22T22:06:39.091 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:06:39.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:06:39.093 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-73014444037 2026-02-22T22:06:39.093 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:06:39.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=73014444037 2026-02-22T22:06:39.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 73014444037' 2026-02-22T22:06:39.096 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 73014444037 2026-02-22T22:06:39.096 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:06:39.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 73014444037 -lt 73014444037 2026-02-22T22:06:39.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:06:39.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182403 2026-02-22T22:06:39.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:06:39.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:06:39.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182403 2026-02-22T22:06:39.297 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:06:39.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182403 2026-02-22T22:06:39.298 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 107374182403 2026-02-22T22:06:39.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182403' 2026-02-22T22:06:39.299 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:06:39.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182403 -lt 107374182403 2026-02-22T22:06:39.497 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:06:39.498 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:06:39.498 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:06:39.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:06:39.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:06:39.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:06:39.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:06:39.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:06:39.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:06:39.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:06:39.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:06:39.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:06:39.950 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:06:39.950 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:06:39.951 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:06:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:06:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:06:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:06:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_1: local shard_id=1 2026-02-22T22:06:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2026-02-22T22:06:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-22T22:06:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2026-02-22T22:06:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2026-02-22T22:06:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2026-02-22T22:06:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2026-02-22T22:06:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2026-02-22T22:06:40.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-71671-1-10 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-71671-1-10 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-71671-1-10 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:06:40.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-71671-1-10 td/test-erasure-eio/ORIGINAL 2026-02-22T22:06:40.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-71671-1-10 td/test-erasure-eio 1 10 set 2026-02-22T22:06:40.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-71671-1-10 2026-02-22T22:06:40.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-71671-1-10 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-71671-1-10 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-71671-1-10 2026-02-22T22:06:40.227 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:06:40.428 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:06:40.428 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:06:40.428 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:06:40.428 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:06:40.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2026-02-22T22:06:40.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2026-02-22T22:06:40.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2026-02-22T22:06:40.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2026-02-22T22:06:40.726 INFO:tasks.workunit.client.0.trial034.stderr:noout is set 2026-02-22T22:06:40.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2026-02-22T22:06:40.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2026-02-22T22:06:40.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2026-02-22T22:06:40.740 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records in 2026-02-22T22:06:40.740 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records out 2026-02-22T22:06:40.740 INFO:tasks.workunit.client.0.trial034.stderr:10 bytes copied, 3.0637e-05 s, 326 kB/s 2026-02-22T22:06:40.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-71671-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:06:40.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:06:40.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:06:40.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2026-02-22T22:06:40.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:06:40.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-71671-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:06:40.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:06:40.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:06:40.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2026-02-22T22:06:40.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:06:40.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-22T22:06:40.741 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:06:40.741 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:06:40.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:06:40.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:06:40.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:06:42.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:06:42.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-71671-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:06:42.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:06:42.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:06:42.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2026-02-22T22:06:42.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:06:42.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-22T22:06:42.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-71671-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:06:43.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-22T22:06:43.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:06:43.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:06:43.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2026-02-22T22:06:43.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:06:43.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:06:43.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:06:43.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:06:43.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:06:43.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:06:43.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:06:43.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:06:43.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:06:43.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:06:43.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:06:43.251 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:06:43.251 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:06:43.251 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:06:43.251 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:06:43.251 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:06:43.251 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:06:43.253 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:06:43.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:06:43.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:06:43.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:06:43.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:06:43.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:06:43.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:06:43.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:06:43.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:06:43.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:06:43.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:06:43.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:06:43.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2026-02-22T22:06:43.255 INFO:tasks.workunit.client.0.trial034.stderr:start osd.1 2026-02-22T22:06:43.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:06:43.255 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-22T22:06:43.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2026-02-22T22:06:43.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:06:43.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:06:43.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:06:43.275 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:43.276+0000 7f7fe015e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:43.276 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:43.276+0000 7f7fe015e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:43.277 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:43.276+0000 7f7fe015e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:43.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2026-02-22T22:06:43.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:06:43.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:06:43.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:06:43.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:06:43.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:43.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:06:43.445 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:06:43.445 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:43.446 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:06:43.645 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:44.528 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:44.528+0000 7f7fe015e8c0 -1 Falling back to public interface 2026-02-22T22:06:44.647 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:44.647 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:44.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:06:44.648 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:06:44.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:44.648 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:06:44.845 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:45.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:45.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:45.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:06:45.847 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:06:45.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:45.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:06:46.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:47.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:47.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:47.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:06:47.050 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:06:47.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:47.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:06:47.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:47.795 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:47.796+0000 7f7fe015e8c0 -1 osd.1 33 log_to_monitors true 2026-02-22T22:06:48.248 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:48.248 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:48.248 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:06:48.248 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:06:48.248 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:48.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:06:48.449 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [12,33) [v2:127.0.0.1:6810/4094704847,v1:127.0.0.1:6811/4094704847] [v2:127.0.0.1:6812/4094704847,v1:127.0.0.1:6813/4094704847] exists,up 5b0429d8-014c-4797-98aa-6e2587f9225f 2026-02-22T22:06:48.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:06:48.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:06:48.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:06:48.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:06:48.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:06:48.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:06:48.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:06:48.449 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:06:48.450 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:06:48.450 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:06:48.450 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:06:48.450 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:06:48.450 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:06:48.562 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:06:48.563 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:06:48.563 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:06:48.563 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:06:48.563 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:06:48.563 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:06:48.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:06:48.759 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:06:48.760 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:06:48.760 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:06:48.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:06:48.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:06:48.760 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:06:48.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803789 2026-02-22T22:06:48.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803789 2026-02-22T22:06:48.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789' 2026-02-22T22:06:48.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:06:48.836 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:06:48.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=167503724547 2026-02-22T22:06:48.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 167503724547 2026-02-22T22:06:48.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-167503724547' 2026-02-22T22:06:48.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:06:48.912 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:06:48.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=73014444041 2026-02-22T22:06:48.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 73014444041 2026-02-22T22:06:48.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-167503724547 2-73014444041' 2026-02-22T22:06:48.991 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:06:48.991 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:06:49.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182408 2026-02-22T22:06:49.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182408 2026-02-22T22:06:49.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-167503724547 2-73014444041 3-107374182408' 2026-02-22T22:06:49.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:06:49.067 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803789 2026-02-22T22:06:49.067 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:06:49.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:06:49.068 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803789 2026-02-22T22:06:49.068 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:06:49.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803789 2026-02-22T22:06:49.070 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803789' 2026-02-22T22:06:49.070 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803789 2026-02-22T22:06:49.070 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:06:49.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803787 -lt 25769803789 2026-02-22T22:06:49.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:06:50.273 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:06:50.274 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:06:50.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803789 -lt 25769803789 2026-02-22T22:06:50.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:06:50.467 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-167503724547 2026-02-22T22:06:50.467 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:06:50.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:06:50.468 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-167503724547 2026-02-22T22:06:50.469 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:06:50.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=167503724547 2026-02-22T22:06:50.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 167503724547' 2026-02-22T22:06:50.470 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 167503724547 2026-02-22T22:06:50.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:06:50.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 167503724547 -lt 167503724547 2026-02-22T22:06:50.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:06:50.668 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-73014444041 2026-02-22T22:06:50.668 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:06:50.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:06:50.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-73014444041 2026-02-22T22:06:50.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:06:50.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=73014444041 2026-02-22T22:06:50.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 73014444041' 2026-02-22T22:06:50.671 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 73014444041 2026-02-22T22:06:50.671 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:06:50.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 73014444042 -lt 73014444041 2026-02-22T22:06:50.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:06:50.866 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182408 2026-02-22T22:06:50.867 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:06:50.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:06:50.868 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182408 2026-02-22T22:06:50.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:06:50.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182408 2026-02-22T22:06:50.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182408' 2026-02-22T22:06:50.870 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 107374182408 2026-02-22T22:06:50.870 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:06:51.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182408 -lt 107374182408 2026-02-22T22:06:51.068 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:06:51.068 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:06:51.069 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:06:51.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:06:51.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:06:51.336 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:06:51.336 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:06:51.336 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:06:51.336 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:06:51.337 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:06:51.337 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:06:51.529 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:06:51.530 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:06:51.530 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:06:51.530 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:06:51.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:06:51.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:06:51.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:06:51.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-22T22:06:51.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2026-02-22T22:06:52.051 INFO:tasks.workunit.client.0.trial034.stderr:noout is unset 2026-02-22T22:06:52.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-71671-1-10 2026-02-22T22:06:52.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:06:52.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:06:52.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-71671-1-10 2026-02-22T22:06:52.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:06:52.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:06:52.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-71671-1-10 td/test-erasure-eio/COPY 2026-02-22T22:06:52.088 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:06:52.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:06:52.091 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2026-02-22T22:06:52.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-71671-1-10 td/test-erasure-eio 2 10 set 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-71671-1-10 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-71671-1-10 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:06:52.093 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-71671-1-10 2026-02-22T22:06:52.094 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-71671-1-10 2026-02-22T22:06:52.094 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:06:52.289 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:06:52.289 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:06:52.289 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:06:52.289 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:06:52.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2026-02-22T22:06:52.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2026-02-22T22:06:52.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2026-02-22T22:06:52.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2026-02-22T22:06:52.559 INFO:tasks.workunit.client.0.trial034.stderr:noout is set 2026-02-22T22:06:52.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2026-02-22T22:06:52.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2026-02-22T22:06:52.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2026-02-22T22:06:52.572 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records in 2026-02-22T22:06:52.572 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records out 2026-02-22T22:06:52.573 INFO:tasks.workunit.client.0.trial034.stderr:10 bytes copied, 3.5325e-05 s, 283 kB/s 2026-02-22T22:06:52.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-71671-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:06:52.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:06:52.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:06:52.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2026-02-22T22:06:52.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:06:52.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-71671-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:06:52.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:06:52.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:06:52.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2026-02-22T22:06:52.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:06:52.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-22T22:06:52.574 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:06:52.574 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:06:52.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:06:52.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:06:52.574 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:06:53.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:06:53.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-71671-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:06:53.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:06:53.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:06:53.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2026-02-22T22:06:53.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:06:53.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-22T22:06:53.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-71671-1-10 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:06:55.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-22T22:06:55.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:06:55.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:06:55.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T22:06:55.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:06:55.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:06:55.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:06:55.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:06:55.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:06:55.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:06:55.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:06:55.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:06:55.104 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:06:55.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:06:55.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:06:55.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:06:55.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:06:55.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:06:55.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:06:55.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:06:55.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:06:55.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T22:06:55.106 INFO:tasks.workunit.client.0.trial034.stderr:start osd.2 2026-02-22T22:06:55.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:06:55.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-22T22:06:55.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T22:06:55.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:06:55.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:06:55.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:06:55.121 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:55.120+0000 7fbcc83868c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:55.123 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:55.124+0000 7fbcc83868c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:55.124 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:55.124+0000 7fbcc83868c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:06:55.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T22:06:55.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:06:55.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:06:55.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:06:55.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:06:55.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:55.304 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:06:55.304 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:06:55.304 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:55.304 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:06:55.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:56.364 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:56.364+0000 7fbcc83868c0 -1 Falling back to public interface 2026-02-22T22:06:56.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:56.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:56.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:06:56.503 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:06:56.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:56.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:06:56.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:57.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:57.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:57.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:06:57.698 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:06:57.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:57.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:06:57.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:58.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:06:58.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:06:58.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:06:58.897 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:06:58.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:06:58.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:06:59.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:06:59.602 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:06:59.599+0000 7fbcc83868c0 -1 osd.2 45 log_to_monitors true 2026-02-22T22:07:00.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:00.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:00.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:07:00.092 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:07:00.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:00.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:07:00.286 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 up in weight 1 up_from 49 up_thru 36 down_at 46 last_clean_interval [17,45) [v2:127.0.0.1:6818/1489226509,v1:127.0.0.1:6819/1489226509] [v2:127.0.0.1:6820/1489226509,v1:127.0.0.1:6821/1489226509] exists,up fa1dd089-f822-4d84-b6b5-4516e45037e0 2026-02-22T22:07:00.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:07:00.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:07:00.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:07:00.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:07:00.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:07:00.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:07:00.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:07:00.287 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:07:00.287 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:07:00.287 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:07:00.287 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:07:00.288 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:07:00.288 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:07:00.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:07:00.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:07:00.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:07:00.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:07:00.398 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:07:00.398 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:07:00.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:07:00.594 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:00.594 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:07:00.594 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:07:00.594 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:07:00.595 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:00.595 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:07:00.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803793 2026-02-22T22:07:00.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803793 2026-02-22T22:07:00.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793' 2026-02-22T22:07:00.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:00.669 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:07:00.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=167503724551 2026-02-22T22:07:00.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 167503724551 2026-02-22T22:07:00.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-167503724551' 2026-02-22T22:07:00.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:00.745 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:07:00.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=210453397507 2026-02-22T22:07:00.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 210453397507 2026-02-22T22:07:00.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-167503724551 2-210453397507' 2026-02-22T22:07:00.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:00.822 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:07:00.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182412 2026-02-22T22:07:00.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182412 2026-02-22T22:07:00.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-167503724551 2-210453397507 3-107374182412' 2026-02-22T22:07:00.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:00.896 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803793 2026-02-22T22:07:00.896 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:00.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:07:00.898 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803793 2026-02-22T22:07:00.898 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:00.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803793 2026-02-22T22:07:00.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803793' 2026-02-22T22:07:00.900 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803793 2026-02-22T22:07:00.900 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:07:01.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803791 -lt 25769803793 2026-02-22T22:07:01.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:07:02.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:07:02.101 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:07:02.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803794 -lt 25769803793 2026-02-22T22:07:02.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:02.300 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-167503724551 2026-02-22T22:07:02.300 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:02.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:07:02.302 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-167503724551 2026-02-22T22:07:02.302 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:02.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=167503724551 2026-02-22T22:07:02.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 167503724551' 2026-02-22T22:07:02.303 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 167503724551 2026-02-22T22:07:02.304 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:07:02.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 167503724551 -lt 167503724551 2026-02-22T22:07:02.510 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:02.511 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-210453397507 2026-02-22T22:07:02.511 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:02.512 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:07:02.513 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-210453397507 2026-02-22T22:07:02.513 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:02.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=210453397507 2026-02-22T22:07:02.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 210453397507' 2026-02-22T22:07:02.514 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 210453397507 2026-02-22T22:07:02.515 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:07:02.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 210453397507 -lt 210453397507 2026-02-22T22:07:02.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:02.713 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182412 2026-02-22T22:07:02.714 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:02.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:07:02.715 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182412 2026-02-22T22:07:02.715 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:02.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182412 2026-02-22T22:07:02.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182412' 2026-02-22T22:07:02.716 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 107374182412 2026-02-22T22:07:02.717 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:07:02.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182412 -lt 107374182412 2026-02-22T22:07:02.922 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:07:02.922 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:07:02.922 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:07:03.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:07:03.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:07:03.175 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:07:03.176 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:07:03.176 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:07:03.176 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:07:03.176 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:07:03.176 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:07:03.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:07:03.368 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:07:03.368 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:07:03.368 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:07:03.627 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:07:03.627 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:07:03.627 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:07:03.627 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-22T22:07:03.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2026-02-22T22:07:03.890 INFO:tasks.workunit.client.0.trial034.stderr:noout is unset 2026-02-22T22:07:03.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-71671-1-10 fail 2026-02-22T22:07:03.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:07:03.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:07:03.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-71671-1-10 2026-02-22T22:07:03.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-22T22:07:03.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-22T22:07:03.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-71671-1-10 td/test-erasure-eio/COPY 2026-02-22T22:07:03.925 INFO:tasks.workunit.client.0.trial034.stderr:error getting pool-jerasure/obj-size-71671-1-10: (5) Input/output error 2026-02-22T22:07:03.927 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-22T22:07:03.927 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-22T22:07:03.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:324: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 0 2026-02-22T22:07:03.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-22T22:07:03.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2026-02-22T22:07:03.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2026-02-22T22:07:03.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2026-02-22T22:07:03.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2026-02-22T22:07:03.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2026-02-22T22:07:03.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2026-02-22T22:07:03.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-22T22:07:03.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-71671-1-0 2026-02-22T22:07:03.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-71671-1-0 2026-02-22T22:07:03.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:07:03.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:07:03.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-71671-1-0 2026-02-22T22:07:03.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:07:03.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:07:03.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:07:03.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:07:03.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:07:03.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:07:03.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:07:03.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:07:03.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-71671-1-0 td/test-erasure-eio/ORIGINAL 2026-02-22T22:07:03.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-71671-1-0 td/test-erasure-eio 1 0 set 2026-02-22T22:07:03.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-71671-1-0 2026-02-22T22:07:03.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2026-02-22T22:07:03.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2026-02-22T22:07:03.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2026-02-22T22:07:03.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2026-02-22T22:07:03.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2026-02-22T22:07:03.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2026-02-22T22:07:03.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2026-02-22T22:07:03.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2026-02-22T22:07:03.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2026-02-22T22:07:03.955 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-71671-1-0 2026-02-22T22:07:03.955 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:07:03.955 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-71671-1-0 2026-02-22T22:07:03.956 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-71671-1-0 2026-02-22T22:07:03.956 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:07:04.144 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:07:04.144 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:04.144 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:07:04.144 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:07:04.144 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2026-02-22T22:07:04.144 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2026-02-22T22:07:04.144 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2026-02-22T22:07:04.144 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2026-02-22T22:07:04.396 INFO:tasks.workunit.client.0.trial034.stderr:noout is set 2026-02-22T22:07:04.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2026-02-22T22:07:04.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2026-02-22T22:07:04.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2026-02-22T22:07:04.411 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-71671-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:04.411 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:07:04.411 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:07:04.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2026-02-22T22:07:04.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:07:04.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-71671-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:04.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:07:04.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:07:04.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2026-02-22T22:07:04.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:07:04.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-22T22:07:04.412 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:07:04.412 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:07:04.412 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:07:04.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:07:04.413 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:07:05.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:07:05.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-71671-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:05.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:07:05.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:07:05.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2026-02-22T22:07:05.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:07:05.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-22T22:07:05.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-71671-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:06.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-22T22:07:06.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:07:06.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:07:06.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2026-02-22T22:07:06.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:07:06.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:07:06.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:07:06.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:07:06.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:07:06.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:07:06.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:07:06.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:07:06.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:07:06.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:07:06.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:07:06.894 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:07:06.894 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:07:06.894 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:07:06.895 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:07:06.895 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:07:06.895 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:start osd.1 2026-02-22T22:07:06.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:07:06.897 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-22T22:07:06.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2026-02-22T22:07:06.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:07:06.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:07:06.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:07:06.917 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:06.915+0000 7f3dd790c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:06.918 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:06.919+0000 7f3dd790c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:06.920 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:06.919+0000 7f3dd790c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:07.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2026-02-22T22:07:07.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:07:07.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:07:07.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:07:07.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:07:07.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:07.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:07:07.098 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:07:07.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:07.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:07.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:08.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:08.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:08.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:07:08.297 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:08.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:08.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:08.415 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:08.415+0000 7f3dd790c8c0 -1 Falling back to public interface 2026-02-22T22:07:08.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:09.495 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:09.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:09.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:07:09.496 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:07:09.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:09.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:09.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:10.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:10.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:10.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:07:10.695 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:07:10.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:10.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:10.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:11.427 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:11.427+0000 7f3dd790c8c0 -1 osd.1 52 log_to_monitors true 2026-02-22T22:07:11.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:11.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:11.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:07:11.899 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:07:11.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:11.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:12.094 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 55 last_clean_interval [39,52) [v2:127.0.0.1:6810/2722985758,v1:127.0.0.1:6811/2722985758] [v2:127.0.0.1:6812/2722985758,v1:127.0.0.1:6813/2722985758] exists,up 5b0429d8-014c-4797-98aa-6e2587f9225f 2026-02-22T22:07:12.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:07:12.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:07:12.094 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:07:12.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:07:12.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:07:12.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:07:12.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:07:12.095 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:07:12.096 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:07:12.096 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:07:12.096 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:07:12.096 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:07:12.096 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:07:12.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:07:12.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:07:12.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:07:12.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:07:12.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:07:12.202 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:07:12.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:07:12.395 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:12.395 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:07:12.395 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:07:12.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:07:12.395 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:12.395 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:07:12.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803798 2026-02-22T22:07:12.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803798 2026-02-22T22:07:12.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803798' 2026-02-22T22:07:12.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:12.470 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:07:12.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=249108103171 2026-02-22T22:07:12.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 249108103171 2026-02-22T22:07:12.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803798 1-249108103171' 2026-02-22T22:07:12.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:12.545 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:07:12.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=210453397511 2026-02-22T22:07:12.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 210453397511 2026-02-22T22:07:12.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803798 1-249108103171 2-210453397511' 2026-02-22T22:07:12.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:12.621 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:07:12.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182416 2026-02-22T22:07:12.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182416 2026-02-22T22:07:12.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803798 1-249108103171 2-210453397511 3-107374182416' 2026-02-22T22:07:12.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:12.696 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803798 2026-02-22T22:07:12.696 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:12.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:07:12.698 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803798 2026-02-22T22:07:12.698 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:12.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803798 2026-02-22T22:07:12.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803798' 2026-02-22T22:07:12.699 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803798 2026-02-22T22:07:12.699 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:07:12.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803796 -lt 25769803798 2026-02-22T22:07:12.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:07:13.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:07:13.901 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:07:14.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803796 -lt 25769803798 2026-02-22T22:07:14.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:07:15.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:07:15.100 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:07:15.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803798 -lt 25769803798 2026-02-22T22:07:15.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:15.296 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-249108103171 2026-02-22T22:07:15.296 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:15.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:07:15.298 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-249108103171 2026-02-22T22:07:15.298 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:15.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=249108103171 2026-02-22T22:07:15.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 249108103171' 2026-02-22T22:07:15.300 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 249108103171 2026-02-22T22:07:15.301 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:07:15.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 249108103171 -lt 249108103171 2026-02-22T22:07:15.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:15.498 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-210453397511 2026-02-22T22:07:15.498 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:15.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:07:15.500 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-210453397511 2026-02-22T22:07:15.500 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:15.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=210453397511 2026-02-22T22:07:15.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 210453397511' 2026-02-22T22:07:15.502 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 210453397511 2026-02-22T22:07:15.502 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:07:15.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 210453397511 -lt 210453397511 2026-02-22T22:07:15.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:15.710 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182416 2026-02-22T22:07:15.710 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:15.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:07:15.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182416 2026-02-22T22:07:15.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:15.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182416 2026-02-22T22:07:15.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182416' 2026-02-22T22:07:15.714 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 107374182416 2026-02-22T22:07:15.714 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:07:15.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182417 -lt 107374182416 2026-02-22T22:07:15.925 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:07:15.926 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:07:15.926 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:07:16.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:07:16.190 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:07:16.190 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:07:16.190 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:07:16.190 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:07:16.190 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:07:16.190 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:07:16.190 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:07:16.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:07:16.386 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:07:16.387 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:07:16.387 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:07:16.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:07:16.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:07:16.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:07:16.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-22T22:07:16.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2026-02-22T22:07:16.908 INFO:tasks.workunit.client.0.trial034.stderr:noout is unset 2026-02-22T22:07:16.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-71671-1-0 2026-02-22T22:07:16.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:07:16.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:07:16.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-71671-1-0 2026-02-22T22:07:16.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:07:16.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:07:16.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-71671-1-0 td/test-erasure-eio/COPY 2026-02-22T22:07:16.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:07:16.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:07:16.950 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2026-02-22T22:07:16.953 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2026-02-22T22:07:16.953 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-71671-1-0 td/test-erasure-eio 2 0 set 2026-02-22T22:07:16.953 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-71671-1-0 2026-02-22T22:07:16.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2026-02-22T22:07:16.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2026-02-22T22:07:16.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2026-02-22T22:07:16.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2026-02-22T22:07:16.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2026-02-22T22:07:16.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2026-02-22T22:07:16.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2026-02-22T22:07:16.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2026-02-22T22:07:16.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2026-02-22T22:07:16.954 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-71671-1-0 2026-02-22T22:07:16.955 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:07:16.955 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-71671-1-0 2026-02-22T22:07:16.955 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-71671-1-0 2026-02-22T22:07:16.955 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:07:17.153 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:07:17.153 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:17.153 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:07:17.153 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:07:17.154 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2026-02-22T22:07:17.154 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2026-02-22T22:07:17.154 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2026-02-22T22:07:17.154 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2026-02-22T22:07:17.415 INFO:tasks.workunit.client.0.trial034.stderr:noout is set 2026-02-22T22:07:17.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2026-02-22T22:07:17.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2026-02-22T22:07:17.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2026-02-22T22:07:17.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-71671-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:17.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:07:17.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:07:17.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2026-02-22T22:07:17.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:07:17.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-71671-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:17.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:07:17.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:07:17.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2026-02-22T22:07:17.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:07:17.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-22T22:07:17.429 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:07:17.429 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:07:17.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:07:17.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:07:17.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:07:18.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:07:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-71671-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:07:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:07:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2026-02-22T22:07:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:07:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-22T22:07:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-71671-1-0 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:19.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-22T22:07:19.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:07:19.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:07:19.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T22:07:19.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:07:19.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:07:19.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:07:19.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:07:19.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:07:19.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:07:19.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:07:19.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:07:19.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:07:19.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:07:19.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:07:19.915 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:07:19.915 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:07:19.916 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:07:19.916 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:07:19.916 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:07:19.916 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:07:19.916 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:07:19.916 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:07:19.916 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:07:19.916 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:07:19.916 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:07:19.916 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:07:19.916 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:07:19.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:07:19.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:07:19.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:07:19.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:07:19.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:07:19.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T22:07:19.917 INFO:tasks.workunit.client.0.trial034.stderr:start osd.2 2026-02-22T22:07:19.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:07:19.918 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-22T22:07:19.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T22:07:19.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:07:19.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:07:19.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:07:19.933 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:19.931+0000 7f63694cc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:19.934 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:19.935+0000 7f63694cc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:19.935 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:19.935+0000 7f63694cc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:20.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T22:07:20.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:07:20.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:07:20.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:07:20.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:07:20.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:20.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:07:20.117 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:07:20.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:20.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:07:20.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:21.208 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:21.207+0000 7f63694cc8c0 -1 Falling back to public interface 2026-02-22T22:07:21.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:21.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:21.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:07:21.313 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:21.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:21.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:07:21.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:22.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:22.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:22.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:07:22.505 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:07:22.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:22.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:07:22.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:23.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:23.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:23.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:07:23.702 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:07:23.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:23.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:07:23.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:24.259 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:24.259+0000 7f63694cc8c0 -1 osd.2 63 log_to_monitors true 2026-02-22T22:07:24.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:24.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:24.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:07:24.898 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:07:24.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:24.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:07:25.092 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 up in weight 1 up_from 67 up_thru 55 down_at 64 last_clean_interval [49,63) [v2:127.0.0.1:6818/3658353077,v1:127.0.0.1:6819/3658353077] [v2:127.0.0.1:6820/3658353077,v1:127.0.0.1:6821/3658353077] exists,up fa1dd089-f822-4d84-b6b5-4516e45037e0 2026-02-22T22:07:25.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:07:25.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:07:25.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:07:25.092 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:07:25.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:07:25.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:07:25.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:07:25.093 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:07:25.094 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:07:25.094 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:07:25.094 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:07:25.094 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:07:25.094 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:07:25.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:07:25.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:07:25.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:07:25.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:07:25.202 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:07:25.202 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:07:25.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:07:25.405 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:25.405 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:07:25.405 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:07:25.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:07:25.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:25.405 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:07:25.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803802 2026-02-22T22:07:25.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803802 2026-02-22T22:07:25.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803802' 2026-02-22T22:07:25.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:25.480 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:07:25.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=249108103175 2026-02-22T22:07:25.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 249108103175 2026-02-22T22:07:25.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803802 1-249108103175' 2026-02-22T22:07:25.557 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:25.558 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:07:25.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=287762808835 2026-02-22T22:07:25.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 287762808835 2026-02-22T22:07:25.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803802 1-249108103175 2-287762808835' 2026-02-22T22:07:25.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:25.636 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:07:25.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182421 2026-02-22T22:07:25.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182421 2026-02-22T22:07:25.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803802 1-249108103175 2-287762808835 3-107374182421' 2026-02-22T22:07:25.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:25.710 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803802 2026-02-22T22:07:25.710 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:25.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:07:25.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803802 2026-02-22T22:07:25.712 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:25.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803802 2026-02-22T22:07:25.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803802' 2026-02-22T22:07:25.713 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803802 2026-02-22T22:07:25.714 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:07:25.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803800 -lt 25769803802 2026-02-22T22:07:25.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:07:26.923 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:07:26.923 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:07:27.124 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803803 -lt 25769803802 2026-02-22T22:07:27.124 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:27.125 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-249108103175 2026-02-22T22:07:27.125 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:27.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:07:27.126 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-249108103175 2026-02-22T22:07:27.126 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:27.127 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=249108103175 2026-02-22T22:07:27.127 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 249108103175' 2026-02-22T22:07:27.128 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 249108103175 2026-02-22T22:07:27.128 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:07:27.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 249108103175 -lt 249108103175 2026-02-22T22:07:27.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:27.332 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-287762808835 2026-02-22T22:07:27.333 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:27.334 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:07:27.334 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-287762808835 2026-02-22T22:07:27.335 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:27.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=287762808835 2026-02-22T22:07:27.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 287762808835' 2026-02-22T22:07:27.336 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 287762808835 2026-02-22T22:07:27.337 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:07:27.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 287762808835 -lt 287762808835 2026-02-22T22:07:27.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:27.541 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182421 2026-02-22T22:07:27.541 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:27.542 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:07:27.543 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182421 2026-02-22T22:07:27.543 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:27.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182421 2026-02-22T22:07:27.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182421' 2026-02-22T22:07:27.544 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 107374182421 2026-02-22T22:07:27.545 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:07:27.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182421 -lt 107374182421 2026-02-22T22:07:27.757 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:07:27.757 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:07:27.757 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:07:28.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:07:28.018 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:07:28.018 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:07:28.018 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:07:28.018 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:07:28.018 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:07:28.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:07:28.019 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:07:28.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:07:28.215 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:07:28.215 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:07:28.215 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:07:28.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:07:28.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:07:28.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:07:28.478 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-22T22:07:28.478 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2026-02-22T22:07:28.742 INFO:tasks.workunit.client.0.trial034.stderr:noout is unset 2026-02-22T22:07:28.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-71671-1-0 fail 2026-02-22T22:07:28.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:07:28.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:07:28.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-71671-1-0 2026-02-22T22:07:28.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-22T22:07:28.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-22T22:07:28.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-71671-1-0 td/test-erasure-eio/COPY 2026-02-22T22:07:28.775 INFO:tasks.workunit.client.0.trial034.stderr:error getting pool-jerasure/obj-size-71671-1-0: (5) Input/output error 2026-02-22T22:07:28.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-22T22:07:28.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-22T22:07:28.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:325: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 256 add 2026-02-22T22:07:28.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2026-02-22T22:07:28.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2026-02-22T22:07:28.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2026-02-22T22:07:28.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2026-02-22T22:07:28.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2026-02-22T22:07:28.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2026-02-22T22:07:28.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2026-02-22T22:07:28.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2026-02-22T22:07:28.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-71671-1-256 2026-02-22T22:07:28.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-71671-1-256 2026-02-22T22:07:28.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:07:28.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:07:28.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-71671-1-256 2026-02-22T22:07:28.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:07:28.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:07:28.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:07:28.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:07:28.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:07:28.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:07:28.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:07:28.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:07:28.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-71671-1-256 td/test-erasure-eio/ORIGINAL 2026-02-22T22:07:28.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-71671-1-256 td/test-erasure-eio 1 256 add 2026-02-22T22:07:28.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-71671-1-256 2026-02-22T22:07:28.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2026-02-22T22:07:28.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2026-02-22T22:07:28.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2026-02-22T22:07:28.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2026-02-22T22:07:28.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2026-02-22T22:07:28.803 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2026-02-22T22:07:28.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2026-02-22T22:07:28.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2026-02-22T22:07:28.804 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2026-02-22T22:07:28.804 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-71671-1-256 2026-02-22T22:07:28.804 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:07:28.804 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-71671-1-256 2026-02-22T22:07:28.805 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-71671-1-256 2026-02-22T22:07:28.805 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:07:28.994 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:07:28.994 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:28.994 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:07:28.994 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:07:28.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2026-02-22T22:07:28.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2026-02-22T22:07:28.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2026-02-22T22:07:28.994 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2026-02-22T22:07:29.246 INFO:tasks.workunit.client.0.trial034.stderr:noout is set 2026-02-22T22:07:29.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2026-02-22T22:07:29.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-71671-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:29.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:07:29.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:07:29.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2026-02-22T22:07:29.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:07:29.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-71671-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:29.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:07:29.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:07:29.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2026-02-22T22:07:29.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:07:29.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-22T22:07:29.261 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:07:29.261 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:07:29.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:07:29.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:07:29.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:07:30.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:07:30.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-71671-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:30.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:07:30.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:07:30.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2026-02-22T22:07:30.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:07:30.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-22T22:07:30.571 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-71671-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:31.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-22T22:07:31.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:07:31.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:07:31.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2026-02-22T22:07:31.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:07:31.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:07:31.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:07:31.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:07:31.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:07:31.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:07:31.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:07:31.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:07:31.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:07:31.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:07:31.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:07:31.423 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:07:31.423 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:07:31.423 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:07:31.423 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:07:31.423 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:07:31.423 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:07:31.424 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:07:31.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:07:31.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:07:31.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:07:31.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:07:31.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:07:31.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:07:31.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:07:31.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:07:31.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:07:31.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:07:31.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:07:31.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2026-02-22T22:07:31.426 INFO:tasks.workunit.client.0.trial034.stderr:start osd.1 2026-02-22T22:07:31.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:07:31.426 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-22T22:07:31.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2026-02-22T22:07:31.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:07:31.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:07:31.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:07:31.437 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:31.435+0000 7f0a7eecf8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:31.439 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:31.439+0000 7f0a7eecf8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:31.440 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:31.439+0000 7f0a7eecf8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:31.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2026-02-22T22:07:31.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:07:31.624 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:07:31.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:07:31.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:07:31.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:31.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:07:31.625 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:07:31.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:31.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:31.820 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:32.704 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:32.703+0000 7f0a7eecf8c0 -1 Falling back to public interface 2026-02-22T22:07:32.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:32.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:32.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:07:32.822 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:32.822 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:32.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:33.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:34.023 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:34.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:34.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:07:34.024 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:07:34.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:34.024 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:34.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:35.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:35.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:35.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:07:35.229 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:07:35.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:35.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:35.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:35.797 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:35.795+0000 7f0a7eecf8c0 -1 osd.1 71 log_to_monitors true 2026-02-22T22:07:36.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:36.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:36.426 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:07:36.426 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:07:36.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:36.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:36.618 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 up in weight 1 up_from 76 up_thru 76 down_at 73 last_clean_interval [58,71) [v2:127.0.0.1:6810/4115506508,v1:127.0.0.1:6811/4115506508] [v2:127.0.0.1:6812/4115506508,v1:127.0.0.1:6813/4115506508] exists,up 5b0429d8-014c-4797-98aa-6e2587f9225f 2026-02-22T22:07:36.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:07:36.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:07:36.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:07:36.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:07:36.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:07:36.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:07:36.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:07:36.619 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:07:36.620 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:07:36.620 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:07:36.620 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:07:36.620 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:07:36.620 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:07:36.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:07:36.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:07:36.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:07:36.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:07:36.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:07:36.721 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:07:36.925 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:07:36.926 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:36.926 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:07:36.926 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:07:36.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:07:36.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:36.927 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:07:36.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803807 2026-02-22T22:07:36.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803807 2026-02-22T22:07:36.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803807' 2026-02-22T22:07:36.999 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:36.999 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:07:37.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=326417514499 2026-02-22T22:07:37.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 326417514499 2026-02-22T22:07:37.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803807 1-326417514499' 2026-02-22T22:07:37.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:37.072 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:07:37.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=287762808839 2026-02-22T22:07:37.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 287762808839 2026-02-22T22:07:37.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803807 1-326417514499 2-287762808839' 2026-02-22T22:07:37.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:37.149 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:07:37.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182425 2026-02-22T22:07:37.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182425 2026-02-22T22:07:37.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803807 1-326417514499 2-287762808839 3-107374182425' 2026-02-22T22:07:37.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:37.224 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803807 2026-02-22T22:07:37.224 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:37.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:07:37.226 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803807 2026-02-22T22:07:37.226 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:37.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803807 2026-02-22T22:07:37.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2026-02-22T22:07:37.228 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803807 2026-02-22T22:07:37.228 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:07:37.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803805 -lt 25769803807 2026-02-22T22:07:37.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:07:38.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:07:38.440 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:07:38.635 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803807 -lt 25769803807 2026-02-22T22:07:38.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:38.636 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-326417514499 2026-02-22T22:07:38.636 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:38.637 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:07:38.638 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-326417514499 2026-02-22T22:07:38.638 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:38.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=326417514499 2026-02-22T22:07:38.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 326417514499' 2026-02-22T22:07:38.640 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 326417514499 2026-02-22T22:07:38.640 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:07:38.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 326417514499 -lt 326417514499 2026-02-22T22:07:38.849 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:38.850 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-287762808839 2026-02-22T22:07:38.850 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:38.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:07:38.852 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-287762808839 2026-02-22T22:07:38.852 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:38.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=287762808839 2026-02-22T22:07:38.853 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 287762808839' 2026-02-22T22:07:38.853 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 287762808839 2026-02-22T22:07:38.854 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:07:39.057 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 287762808839 -lt 287762808839 2026-02-22T22:07:39.057 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:39.058 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182425 2026-02-22T22:07:39.058 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:39.059 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:07:39.060 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182425 2026-02-22T22:07:39.060 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:39.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182425 2026-02-22T22:07:39.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182425' 2026-02-22T22:07:39.062 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 107374182425 2026-02-22T22:07:39.062 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:07:39.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182425 -lt 107374182425 2026-02-22T22:07:39.260 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:07:39.260 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:07:39.260 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:07:39.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:07:39.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:07:39.520 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:07:39.520 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:07:39.521 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:07:39.521 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:07:39.521 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:07:39.521 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:07:39.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:07:39.714 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:07:39.714 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:07:39.715 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:07:39.978 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:07:39.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:07:39.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:07:39.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2026-02-22T22:07:39.981 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records in 2026-02-22T22:07:39.981 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records out 2026-02-22T22:07:39.981 INFO:tasks.workunit.client.0.trial034.stderr:256 bytes copied, 2.3603e-05 s, 10.8 MB/s 2026-02-22T22:07:39.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-71671-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:39.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:07:39.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:07:39.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2026-02-22T22:07:39.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:07:39.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-71671-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:39.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:07:39.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:07:39.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2026-02-22T22:07:39.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:07:39.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-22T22:07:39.982 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:07:39.982 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:07:39.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:07:39.982 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:07:39.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:07:41.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:07:41.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-71671-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:41.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:07:41.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:07:41.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2026-02-22T22:07:41.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:07:41.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-22T22:07:41.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-71671-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:42.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-22T22:07:42.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:07:42.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:07:42.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2026-02-22T22:07:42.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:07:42.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:07:42.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:07:42.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:07:42.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:07:42.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:07:42.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:07:42.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:07:42.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:07:42.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:07:42.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:07:42.471 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:07:42.471 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:07:42.471 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:07:42.473 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:07:42.474 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:07:42.474 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:07:42.474 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:07:42.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:07:42.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:07:42.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:07:42.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:07:42.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:07:42.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:07:42.474 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:07:42.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:07:42.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:07:42.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:07:42.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:07:42.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2026-02-22T22:07:42.475 INFO:tasks.workunit.client.0.trial034.stderr:start osd.1 2026-02-22T22:07:42.476 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:07:42.476 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-22T22:07:42.476 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2026-02-22T22:07:42.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:07:42.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:07:42.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:07:42.495 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:42.494+0000 7f17f72918c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:42.497 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:42.494+0000 7f17f72918c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:42.498 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:42.494+0000 7f17f72918c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:42.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2026-02-22T22:07:42.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:07:42.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:07:42.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:07:42.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:07:42.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:42.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:07:42.672 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:07:42.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:42.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:42.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:43.724 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:43.722+0000 7f17f72918c0 -1 Falling back to public interface 2026-02-22T22:07:43.875 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:43.875 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:43.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:07:43.876 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:43.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:43.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:44.072 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:45.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:45.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:45.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:07:45.075 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:07:45.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:45.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:45.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:46.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:46.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:46.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:07:46.277 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:07:46.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:46.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:46.478 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:46.791 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:46.790+0000 7f17f72918c0 -1 osd.1 77 log_to_monitors true 2026-02-22T22:07:47.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:47.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:47.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:07:47.480 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:07:47.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:47.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:07:47.676 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 up in weight 1 up_from 81 up_thru 81 down_at 78 last_clean_interval [76,77) [v2:127.0.0.1:6810/1021984953,v1:127.0.0.1:6811/1021984953] [v2:127.0.0.1:6812/1021984953,v1:127.0.0.1:6813/1021984953] exists,up 5b0429d8-014c-4797-98aa-6e2587f9225f 2026-02-22T22:07:47.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:07:47.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:07:47.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:07:47.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:07:47.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:07:47.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:07:47.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:07:47.677 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:07:47.678 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:07:47.678 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:07:47.678 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:07:47.678 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:07:47.678 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:07:47.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:07:47.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:07:47.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:07:47.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:07:47.783 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:07:47.784 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:07:47.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:07:47.979 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:47.979 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:07:47.979 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:07:47.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:07:47.979 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:47.979 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:07:48.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803811 2026-02-22T22:07:48.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803811 2026-02-22T22:07:48.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803811' 2026-02-22T22:07:48.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:48.056 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:07:48.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=347892350979 2026-02-22T22:07:48.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 347892350979 2026-02-22T22:07:48.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803811 1-347892350979' 2026-02-22T22:07:48.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:48.132 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:07:48.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=287762808843 2026-02-22T22:07:48.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 287762808843 2026-02-22T22:07:48.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803811 1-347892350979 2-287762808843' 2026-02-22T22:07:48.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:07:48.207 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:07:48.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182429 2026-02-22T22:07:48.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182429 2026-02-22T22:07:48.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803811 1-347892350979 2-287762808843 3-107374182429' 2026-02-22T22:07:48.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:48.283 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803811 2026-02-22T22:07:48.284 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:48.284 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:07:48.285 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803811 2026-02-22T22:07:48.285 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:48.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803811 2026-02-22T22:07:48.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2026-02-22T22:07:48.287 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803811 2026-02-22T22:07:48.287 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:07:48.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803811 -lt 25769803811 2026-02-22T22:07:48.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:48.490 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-347892350979 2026-02-22T22:07:48.490 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:48.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:07:48.492 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-347892350979 2026-02-22T22:07:48.492 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:48.493 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=347892350979 2026-02-22T22:07:48.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 347892350979' 2026-02-22T22:07:48.494 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 347892350979 2026-02-22T22:07:48.494 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:07:48.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 1 -lt 347892350979 2026-02-22T22:07:48.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:07:49.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:07:49.687 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:07:49.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 1 -lt 347892350979 2026-02-22T22:07:49.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:07:50.890 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:07:50.891 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:07:51.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 347892350979 -lt 347892350979 2026-02-22T22:07:51.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:51.083 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-287762808843 2026-02-22T22:07:51.083 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:51.084 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:07:51.085 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-287762808843 2026-02-22T22:07:51.085 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:51.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=287762808843 2026-02-22T22:07:51.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 287762808843' 2026-02-22T22:07:51.087 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 287762808843 2026-02-22T22:07:51.087 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:07:51.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 287762808844 -lt 287762808843 2026-02-22T22:07:51.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:07:51.286 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182429 2026-02-22T22:07:51.286 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:07:51.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:07:51.288 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182429 2026-02-22T22:07:51.288 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:07:51.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182429 2026-02-22T22:07:51.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182429' 2026-02-22T22:07:51.289 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 107374182429 2026-02-22T22:07:51.290 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:07:51.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182430 -lt 107374182429 2026-02-22T22:07:51.503 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:07:51.504 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:07:51.504 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:07:51.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:07:51.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:07:51.766 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:07:51.766 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:07:51.766 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:07:51.766 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:07:51.766 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:07:51.766 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:07:51.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:07:51.965 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:07:51.965 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:07:51.965 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:07:52.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:07:52.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:07:52.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:07:52.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-22T22:07:52.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2026-02-22T22:07:52.493 INFO:tasks.workunit.client.0.trial034.stderr:noout is unset 2026-02-22T22:07:52.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-71671-1-256 2026-02-22T22:07:52.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:07:52.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:07:52.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-71671-1-256 2026-02-22T22:07:52.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:07:52.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:07:52.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-71671-1-256 td/test-erasure-eio/COPY 2026-02-22T22:07:52.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:07:52.531 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:07:52.533 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2026-02-22T22:07:52.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2026-02-22T22:07:52.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-71671-1-256 td/test-erasure-eio 2 256 add 2026-02-22T22:07:52.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-71671-1-256 2026-02-22T22:07:52.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2026-02-22T22:07:52.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2026-02-22T22:07:52.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2026-02-22T22:07:52.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2026-02-22T22:07:52.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2026-02-22T22:07:52.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2026-02-22T22:07:52.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2026-02-22T22:07:52.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2026-02-22T22:07:52.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2026-02-22T22:07:52.536 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-71671-1-256 2026-02-22T22:07:52.537 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:07:52.537 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-71671-1-256 2026-02-22T22:07:52.537 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-71671-1-256 2026-02-22T22:07:52.537 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:07:52.739 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:07:52.739 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:52.739 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:07:52.739 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:07:52.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2026-02-22T22:07:52.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2026-02-22T22:07:52.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2026-02-22T22:07:52.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2026-02-22T22:07:52.999 INFO:tasks.workunit.client.0.trial034.stderr:noout is set 2026-02-22T22:07:53.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2026-02-22T22:07:53.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-71671-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:53.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:07:53.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:07:53.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2026-02-22T22:07:53.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:07:53.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-71671-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:53.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:07:53.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:07:53.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2026-02-22T22:07:53.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:07:53.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-22T22:07:53.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:07:53.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:07:53.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:07:53.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:07:53.015 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:07:54.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:07:54.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-71671-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:54.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:07:54.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:07:54.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2026-02-22T22:07:54.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:07:54.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-22T22:07:54.324 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-71671-1-256 get-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:07:55.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-22T22:07:55.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:07:55.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:07:55.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T22:07:55.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:07:55.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:07:55.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:07:55.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:07:55.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:07:55.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:07:55.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:07:55.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:07:55.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:07:55.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:07:55.177 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:07:55.177 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:07:55.178 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:07:55.178 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:07:55.180 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:07:55.180 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:07:55.180 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:07:55.180 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:07:55.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:07:55.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:07:55.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:07:55.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:07:55.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:07:55.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:07:55.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:07:55.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:07:55.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:07:55.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:07:55.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:07:55.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T22:07:55.181 INFO:tasks.workunit.client.0.trial034.stderr:start osd.2 2026-02-22T22:07:55.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:07:55.181 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-22T22:07:55.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T22:07:55.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:07:55.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:07:55.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:07:55.200 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:55.198+0000 7fb4275e68c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:55.201 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:55.198+0000 7fb4275e68c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:55.202 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:55.202+0000 7fb4275e68c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:07:55.381 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T22:07:55.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:07:55.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:07:55.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:07:55.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:07:55.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:55.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:07:55.382 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:07:55.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:55.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:07:55.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:56.424 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:56.422+0000 7fb4275e68c0 -1 Falling back to public interface 2026-02-22T22:07:56.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:56.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:56.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:07:56.584 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:07:56.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:56.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:07:56.784 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:57.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:57.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:57.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:07:57.786 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:07:57.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:57.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:07:57.981 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:58.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:07:58.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:07:58.983 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:07:58.983 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:07:58.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:07:58.984 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:07:59.183 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:07:59.483 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:07:59.482+0000 7fb4275e68c0 -1 osd.2 85 log_to_monitors true 2026-02-22T22:08:00.185 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:00.185 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:00.185 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:08:00.185 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:08:00.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:00.186 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:08:00.382 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 up in weight 1 up_from 90 up_thru 78 down_at 87 last_clean_interval [67,85) [v2:127.0.0.1:6818/4280766414,v1:127.0.0.1:6819/4280766414] [v2:127.0.0.1:6820/4280766414,v1:127.0.0.1:6821/4280766414] exists,up fa1dd089-f822-4d84-b6b5-4516e45037e0 2026-02-22T22:08:00.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:08:00.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:08:00.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:08:00.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:08:00.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:08:00.382 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:08:00.383 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:08:00.383 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:08:00.383 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:08:00.384 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:08:00.384 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:08:00.384 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:08:00.384 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:08:00.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:08:00.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:08:00.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:08:00.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:08:00.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:08:00.497 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:08:00.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:08:00.694 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:08:00.694 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:08:00.694 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:08:00.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:08:00.694 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:08:00.694 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:08:00.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803815 2026-02-22T22:08:00.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803815 2026-02-22T22:08:00.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803815' 2026-02-22T22:08:00.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:08:00.770 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:08:00.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=347892350983 2026-02-22T22:08:00.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 347892350983 2026-02-22T22:08:00.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803815 1-347892350983' 2026-02-22T22:08:00.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:08:00.844 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:08:00.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=386547056643 2026-02-22T22:08:00.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 386547056643 2026-02-22T22:08:00.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803815 1-347892350983 2-386547056643' 2026-02-22T22:08:00.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:08:00.921 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:08:00.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182434 2026-02-22T22:08:00.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182434 2026-02-22T22:08:00.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803815 1-347892350983 2-386547056643 3-107374182434' 2026-02-22T22:08:00.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:08:00.996 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803815 2026-02-22T22:08:00.997 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:08:00.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:08:00.998 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803815 2026-02-22T22:08:00.998 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:08:01.000 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803815 2026-02-22T22:08:01.000 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803815' 2026-02-22T22:08:01.000 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803815 2026-02-22T22:08:01.000 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:08:01.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803813 -lt 25769803815 2026-02-22T22:08:01.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:08:02.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:08:02.201 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:08:02.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803816 -lt 25769803815 2026-02-22T22:08:02.394 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:08:02.395 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-347892350983 2026-02-22T22:08:02.395 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:08:02.396 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:08:02.397 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-347892350983 2026-02-22T22:08:02.397 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:08:02.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=347892350983 2026-02-22T22:08:02.399 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 347892350983' 2026-02-22T22:08:02.399 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 347892350983 2026-02-22T22:08:02.399 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:08:02.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 347892350984 -lt 347892350983 2026-02-22T22:08:02.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:08:02.598 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-386547056643 2026-02-22T22:08:02.598 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:08:02.599 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:08:02.600 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-386547056643 2026-02-22T22:08:02.600 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:08:02.601 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=386547056643 2026-02-22T22:08:02.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 386547056643' 2026-02-22T22:08:02.602 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 386547056643 2026-02-22T22:08:02.602 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:08:02.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 386547056643 -lt 386547056643 2026-02-22T22:08:02.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:08:02.808 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182434 2026-02-22T22:08:02.808 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:08:02.809 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:08:02.810 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182434 2026-02-22T22:08:02.810 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:08:02.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182434 2026-02-22T22:08:02.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182434' 2026-02-22T22:08:02.811 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 107374182434 2026-02-22T22:08:02.812 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:08:03.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182434 -lt 107374182434 2026-02-22T22:08:03.018 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:08:03.018 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:08:03.018 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:08:03.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:08:03.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:08:03.277 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:08:03.277 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:08:03.277 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:08:03.277 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:08:03.278 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:08:03.278 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:08:03.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:08:03.478 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:08:03.478 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:08:03.478 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:08:03.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:08:03.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:08:03.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:08:03.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2026-02-22T22:08:03.741 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records in 2026-02-22T22:08:03.741 INFO:tasks.workunit.client.0.trial034.stderr:1+0 records out 2026-02-22T22:08:03.741 INFO:tasks.workunit.client.0.trial034.stderr:256 bytes copied, 2.4255e-05 s, 10.6 MB/s 2026-02-22T22:08:03.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-71671-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:08:03.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:08:03.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:08:03.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2026-02-22T22:08:03.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:08:03.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-71671-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:08:03.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:08:03.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:08:03.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2026-02-22T22:08:03.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:08:03.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-22T22:08:03.742 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:08:03.742 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:08:03.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:08:03.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:08:03.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:08:05.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:08:05.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-71671-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:08:05.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:08:05.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:08:05.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2026-02-22T22:08:05.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:08:05.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-22T22:08:05.052 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-71671-1-256 set-bytes td/test-erasure-eio/CORRUPT 2026-02-22T22:08:06.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-22T22:08:06.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:08:06.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:08:06.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T22:08:06.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:08:06.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:08:06.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:08:06.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:08:06.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:08:06.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:08:06.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:08:06.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:08:06.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:08:06.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:08:06.224 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:08:06.228 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:08:06.228 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:08:06.229 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:08:06.229 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:08:06.229 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:06.229 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:06.229 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:06.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:06.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:08:06.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:08:06.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:08:06.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:08:06.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:08:06.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:08:06.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:08:06.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:08:06.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:08:06.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:08:06.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T22:08:06.230 INFO:tasks.workunit.client.0.trial034.stderr:start osd.2 2026-02-22T22:08:06.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:08:06.231 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-22T22:08:06.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T22:08:06.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:08:06.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:08:06.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:08:06.247 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:06.246+0000 7fb03f19f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:06.249 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:06.246+0000 7fb03f19f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:06.250 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:06.246+0000 7fb03f19f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:06.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T22:08:06.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:08:06.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:08:06.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:08:06.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:08:06.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:06.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:08:06.423 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:08:06.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:06.423 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:08:06.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:07.476 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:07.474+0000 7fb03f19f8c0 -1 Falling back to public interface 2026-02-22T22:08:07.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:07.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:07.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:08:07.622 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:08:07.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:07.622 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:08:07.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:08.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:08.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:08.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:08:08.825 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:08:08.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:08.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:08:09.029 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:10.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:10.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:10.030 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:08:10.031 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:08:10.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:10.031 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:08:10.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:10.538 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:10.534+0000 7fb03f19f8c0 -1 osd.2 91 log_to_monitors true 2026-02-22T22:08:11.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:11.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:11.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:08:11.231 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:08:11.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:11.231 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:08:11.433 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 up in weight 1 up_from 95 up_thru 78 down_at 92 last_clean_interval [90,91) [v2:127.0.0.1:6818/485379170,v1:127.0.0.1:6819/485379170] [v2:127.0.0.1:6820/485379170,v1:127.0.0.1:6821/485379170] exists,up fa1dd089-f822-4d84-b6b5-4516e45037e0 2026-02-22T22:08:11.433 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:08:11.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:08:11.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:08:11.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:08:11.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:08:11.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:08:11.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:08:11.434 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:08:11.435 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:08:11.435 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:08:11.435 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:08:11.435 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:08:11.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:08:11.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:08:11.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:08:11.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:08:11.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:08:11.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:08:11.541 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:08:11.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:08:11.740 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:08:11.740 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:08:11.740 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:08:11.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:08:11.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:08:11.740 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:08:11.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803820 2026-02-22T22:08:11.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803820 2026-02-22T22:08:11.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803820' 2026-02-22T22:08:11.819 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:08:11.819 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:08:11.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=347892350988 2026-02-22T22:08:11.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 347892350988 2026-02-22T22:08:11.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803820 1-347892350988' 2026-02-22T22:08:11.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:08:11.896 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:08:11.972 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=408021893123 2026-02-22T22:08:11.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 408021893123 2026-02-22T22:08:11.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803820 1-347892350988 2-408021893123' 2026-02-22T22:08:11.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:08:11.973 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:08:12.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182438 2026-02-22T22:08:12.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182438 2026-02-22T22:08:12.047 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803820 1-347892350988 2-408021893123 3-107374182438' 2026-02-22T22:08:12.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:08:12.048 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803820 2026-02-22T22:08:12.048 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:08:12.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:08:12.050 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803820 2026-02-22T22:08:12.050 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:08:12.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803820 2026-02-22T22:08:12.051 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803820' 2026-02-22T22:08:12.051 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803820 2026-02-22T22:08:12.052 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:08:12.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803820 -lt 25769803820 2026-02-22T22:08:12.254 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:08:12.255 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-347892350988 2026-02-22T22:08:12.255 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:08:12.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:08:12.257 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-347892350988 2026-02-22T22:08:12.257 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:08:12.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=347892350988 2026-02-22T22:08:12.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 347892350988' 2026-02-22T22:08:12.259 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 347892350988 2026-02-22T22:08:12.259 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:08:12.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 347892350988 -lt 347892350988 2026-02-22T22:08:12.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:08:12.459 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-408021893123 2026-02-22T22:08:12.460 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:08:12.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:08:12.461 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-408021893123 2026-02-22T22:08:12.462 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:08:12.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=408021893123 2026-02-22T22:08:12.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 408021893123' 2026-02-22T22:08:12.463 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 408021893123 2026-02-22T22:08:12.463 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:08:12.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 408021893123 -lt 408021893123 2026-02-22T22:08:12.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:08:12.654 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182438 2026-02-22T22:08:12.655 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:08:12.655 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:08:12.656 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182438 2026-02-22T22:08:12.656 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:08:12.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182438 2026-02-22T22:08:12.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182438' 2026-02-22T22:08:12.658 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 107374182438 2026-02-22T22:08:12.658 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:08:12.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182438 -lt 107374182438 2026-02-22T22:08:12.870 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:08:12.870 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:08:12.870 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:08:13.131 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:08:13.132 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:08:13.132 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:08:13.132 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:08:13.132 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:08:13.132 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:08:13.132 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:08:13.132 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:08:13.326 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:08:13.326 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:08:13.326 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:08:13.326 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:08:13.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:08:13.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:08:13.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:08:13.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2026-02-22T22:08:13.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2026-02-22T22:08:13.853 INFO:tasks.workunit.client.0.trial034.stderr:noout is unset 2026-02-22T22:08:13.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-71671-1-256 fail 2026-02-22T22:08:13.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:08:13.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:08:13.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-71671-1-256 2026-02-22T22:08:13.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-22T22:08:13.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-22T22:08:13.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-71671-1-256 td/test-erasure-eio/COPY 2026-02-22T22:08:13.891 INFO:tasks.workunit.client.0.trial034.stderr:error getting pool-jerasure/obj-size-71671-1-256: (5) Input/output error 2026-02-22T22:08:13.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-22T22:08:13.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2026-02-22T22:08:13.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:326: TEST_rados_get_bad_size_shard_1: delete_erasure_coded_pool pool-jerasure 2026-02-22T22:08:13.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:08:13.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T22:08:14.101 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T22:08:14.110 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T22:08:14.410 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T22:08:14.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T22:08:14.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:08:14.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:08:14.420 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:08:14.420 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:08:14.420 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:08:14.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:08:14.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:08:14.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:08:14.732 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:08:14.732 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:08:14.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:08:14.734 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:08:14.735 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:08:14.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:08:14.736 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:08:14.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:08:14.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:08:14.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:08:14.738 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:08:14.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:08:14.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:08:14.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:08:14.765 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:08:14.765 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:14.765 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:14.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:08:14.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:08:14.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:08:14.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T22:08:14.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T22:08:14.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T22:08:14.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T22:08:14.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:08:14.768 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:08:14.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:08:14.769 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:08:14.769 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:08:14.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:08:14.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:08:14.769 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:08:14.770 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:08:14.771 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:08:14.771 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:08:14.772 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:08:14.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:08:14.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:08:14.773 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:08:14.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:08:14.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:08:14.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:08:14.775 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:08:14.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:08:14.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:08:14.777 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:08:14.779 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:08:14.779 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:14.779 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:14.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:08:14.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:08:14.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:08:14.780 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T22:08:14.782 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:08:14.782 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:14.782 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:14.782 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T22:08:14.785 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:08:14.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:08:14.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:08:14.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T22:08:14.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T22:08:14.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T22:08:14.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:08:14.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:08:14.785 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:08:14.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T22:08:14.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T22:08:14.806 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:08:14.806 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:08:14.806 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:08:14.806 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:08:14.806 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:14.807 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:14.807 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:14.807 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:08:14.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:08:14.835 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:08:14.835 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:08:14.835 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:08:14.835 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:08:14.835 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:08:14.836 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:08:14.837 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:08:14.837 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:08:14.837 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:08:14.837 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:14.837 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:14.837 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:08:14.837 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:08:14.837 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T22:08:14.891 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:08:14.891 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:08:14.891 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:08:14.891 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:08:14.892 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:08:14.892 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:08:14.892 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:08:14.892 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:08:14.892 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:08:14.892 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:14.892 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:14.893 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:08:14.893 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:08:14.893 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T22:08:14.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T22:08:14.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T22:08:14.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:08:14.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:08:14.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:08:14.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T22:08:14.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:08:15.058 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:08:15.059 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:08:15.059 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:08:15.059 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:08:15.059 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:15.059 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:15.059 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:15.060 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:08:15.060 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:08:15.079 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T22:08:15.079 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:08:15.184 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T22:08:15.192 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:08:16.195 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T22:08:16.195 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:08:16.195 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:08:16.195 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:08:16.195 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:16.195 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:16.195 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:08:16.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T22:08:16.196 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T22:08:16.244 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:08:16.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T22:08:16.252 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:08:14.822+0000 7f4e9cbe6ec0 0 load: jerasure load: lrc 2026-02-22T22:08:16.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_0 td/test-erasure-eio 2026-02-22T22:08:16.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:255: TEST_rados_get_subread_eio_shard_0: local dir=td/test-erasure-eio 2026-02-22T22:08:16.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:256: TEST_rados_get_subread_eio_shard_0: setup_osds 4 2026-02-22T22:08:16.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-22T22:08:16.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T22:08:16.254 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-22T22:08:16.254 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-22T22:08:16.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:08:16.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T22:08:16.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:08:16.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:08:16.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:08:16.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:08:16.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T22:08:16.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:08:16.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:08:16.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:08:16.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:08:16.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T22:08:16.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T22:08:16.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:08:16.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:08:16.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:08:16.259 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:08:16.259 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:08:16.259 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:08:16.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:08:16.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:08:16.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:08:16.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:08:16.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T22:08:16.261 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:08:16.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=559971cc-0d79-45e3-8d18-54432929e7e4 2026-02-22T22:08:16.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 559971cc-0d79-45e3-8d18-54432929e7e4' 2026-02-22T22:08:16.262 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 559971cc-0d79-45e3-8d18-54432929e7e4 2026-02-22T22:08:16.262 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:08:16.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQfptpwQ8hEBAAK2Q6Rej9lADVL/accb+Qjw== 2026-02-22T22:08:16.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQfptpwQ8hEBAAK2Q6Rej9lADVL/accb+Qjw=="}' 2026-02-22T22:08:16.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 559971cc-0d79-45e3-8d18-54432929e7e4 -i td/test-erasure-eio/0/new.json 2026-02-22T22:08:16.376 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:08:16.384 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T22:08:16.385 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQfptpwQ8hEBAAK2Q6Rej9lADVL/accb+Qjw== --osd-uuid 559971cc-0d79-45e3-8d18-54432929e7e4 2026-02-22T22:08:16.400 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:16.397+0000 7f4902d528c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:16.401 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:16.397+0000 7f4902d528c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:16.402 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:16.401+0000 7f4902d528c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:16.403 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:16.401+0000 7f4902d528c0 -1 bdev(0x555b1c1d7c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:08:16.403 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:16.401+0000 7f4902d528c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T22:08:19.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T22:08:19.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:08:19.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:08:19.021 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T22:08:19.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:08:19.294 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T22:08:19.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:08:19.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:08:19.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:08:19.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:08:19.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:08:19.318 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:19.317+0000 7fa8478eb8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:19.320 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:19.317+0000 7fa8478eb8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:19.321 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:19.317+0000 7fa8478eb8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:19.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:08:19.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:08:19.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:08:19.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:08:19.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:08:19.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:19.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:08:19.491 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:08:19.492 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:19.492 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:08:19.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:20.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:20.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:20.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:08:20.692 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:08:20.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:20.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:08:20.832 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:20.829+0000 7fa8478eb8c0 -1 Falling back to public interface 2026-02-22T22:08:20.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:21.890 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:21.890 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:21.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:08:21.891 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:08:21.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:21.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:08:22.089 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:23.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:23.090 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:23.090 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:08:23.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:08:23.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:23.091 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:08:23.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:23.811 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:23.809+0000 7fa8478eb8c0 -1 osd.0 0 log_to_monitors true 2026-02-22T22:08:24.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:24.290 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:24.290 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:08:24.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:08:24.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:24.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:08:24.484 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/463942539,v1:127.0.0.1:6803/463942539] [v2:127.0.0.1:6804/463942539,v1:127.0.0.1:6805/463942539] exists,up 559971cc-0d79-45e3-8d18-54432929e7e4 2026-02-22T22:08:24.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:08:24.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:08:24.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:08:24.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:08:24.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T22:08:24.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:08:24.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:08:24.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:08:24.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:08:24.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:08:24.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:08:24.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:08:24.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:08:24.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:08:24.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:08:24.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:08:24.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:08:24.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:08:24.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:08:24.489 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:08:24.490 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:08:24.490 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:08:24.490 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:08:24.490 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:24.490 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:24.490 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:24.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:24.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:08:24.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:08:24.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:08:24.490 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:08:24.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:08:24.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:08:24.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:08:24.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:08:24.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:08:24.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:08:24.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:08:24.491 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:08:24.492 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:08:24.492 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=76ec9b3f-f002-41cd-bf4e-f940147cbca3 2026-02-22T22:08:24.492 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 76ec9b3f-f002-41cd-bf4e-f940147cbca3' 2026-02-22T22:08:24.493 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 76ec9b3f-f002-41cd-bf4e-f940147cbca3 2026-02-22T22:08:24.493 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:08:24.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBYfptpIz7pHRAAkAVtQmWrmm9fUyJiVsIC/w== 2026-02-22T22:08:24.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBYfptpIz7pHRAAkAVtQmWrmm9fUyJiVsIC/w=="}' 2026-02-22T22:08:24.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 76ec9b3f-f002-41cd-bf4e-f940147cbca3 -i td/test-erasure-eio/1/new.json 2026-02-22T22:08:24.705 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:08:24.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T22:08:24.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBYfptpIz7pHRAAkAVtQmWrmm9fUyJiVsIC/w== --osd-uuid 76ec9b3f-f002-41cd-bf4e-f940147cbca3 2026-02-22T22:08:24.738 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:24.733+0000 7fadec7cd8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:24.740 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:24.737+0000 7fadec7cd8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:24.741 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:24.737+0000 7fadec7cd8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:24.741 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:24.737+0000 7fadec7cd8c0 -1 bdev(0x55a803807c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:08:24.742 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:24.737+0000 7fadec7cd8c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T22:08:27.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T22:08:27.020 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:08:27.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:08:27.021 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T22:08:27.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:08:27.297 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T22:08:27.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:08:27.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:08:27.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:08:27.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:08:27.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:08:27.320 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:27.317+0000 7f7e710858c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:27.322 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:27.317+0000 7f7e710858c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:27.323 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:27.321+0000 7f7e710858c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:27.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:08:27.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:08:27.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:08:27.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:08:27.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:08:27.498 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:08:27.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:27.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:08:27.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:27.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:08:27.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:28.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:28.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:28.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:08:28.700 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:08:28.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:28.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:08:28.860 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:28.857+0000 7f7e710858c0 -1 Falling back to public interface 2026-02-22T22:08:28.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:29.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:29.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:29.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:08:29.900 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:08:29.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:29.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:08:30.097 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:31.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:31.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:31.099 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:08:31.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:08:31.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:31.100 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:08:31.300 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:31.877 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:31.873+0000 7f7e710858c0 -1 osd.1 0 log_to_monitors true 2026-02-22T22:08:32.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:32.301 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:32.302 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:08:32.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:08:32.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:32.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:08:32.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:33.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:33.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:33.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:08:33.503 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:08:33.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:33.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:08:33.696 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2893731093,v1:127.0.0.1:6811/2893731093] [v2:127.0.0.1:6812/2893731093,v1:127.0.0.1:6813/2893731093] exists,up 76ec9b3f-f002-41cd-bf4e-f940147cbca3 2026-02-22T22:08:33.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:08:33.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:08:33.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:08:33.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:08:33.698 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:33.699 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:33.699 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:33.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:33.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:08:33.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=90fffe8d-6606-4437-821c-dccc4185c64e 2026-02-22T22:08:33.701 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 90fffe8d-6606-4437-821c-dccc4185c64e 2026-02-22T22:08:33.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 90fffe8d-6606-4437-821c-dccc4185c64e' 2026-02-22T22:08:33.701 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:08:33.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhfptpotWSKhAAGcEm5GixEEpBm9x+Ds5mcg== 2026-02-22T22:08:33.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhfptpotWSKhAAGcEm5GixEEpBm9x+Ds5mcg=="}' 2026-02-22T22:08:33.713 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 90fffe8d-6606-4437-821c-dccc4185c64e -i td/test-erasure-eio/2/new.json 2026-02-22T22:08:33.920 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:08:33.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T22:08:33.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhfptpotWSKhAAGcEm5GixEEpBm9x+Ds5mcg== --osd-uuid 90fffe8d-6606-4437-821c-dccc4185c64e 2026-02-22T22:08:33.947 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:33.945+0000 7f27186308c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:33.948 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:33.945+0000 7f27186308c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:33.949 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:33.945+0000 7f27186308c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:33.950 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:33.945+0000 7f27186308c0 -1 bdev(0x55eaebe85c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:08:33.950 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:33.949+0000 7f27186308c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T22:08:36.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T22:08:36.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:08:36.236 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:08:36.237 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T22:08:36.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:08:36.517 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T22:08:36.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:08:36.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:08:36.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:08:36.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:08:36.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:08:36.540 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:36.537+0000 7f659c0848c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:36.542 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:36.537+0000 7f659c0848c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:36.543 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:36.541+0000 7f659c0848c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:36.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:08:36.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:08:36.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:08:36.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:08:36.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:08:36.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:36.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:08:36.716 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:08:36.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:36.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:08:36.913 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:37.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:37.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:37.914 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:08:37.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:08:37.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:37.915 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:08:38.052 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:38.049+0000 7f659c0848c0 -1 Falling back to public interface 2026-02-22T22:08:38.113 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:39.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:39.114 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:39.115 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:08:39.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:08:39.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:39.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:08:39.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:40.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:40.313 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:40.313 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:08:40.314 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:08:40.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:40.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:08:40.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:41.045 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:41.041+0000 7f659c0848c0 -1 osd.2 0 log_to_monitors true 2026-02-22T22:08:41.516 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:41.516 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:41.517 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:08:41.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:08:41.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:41.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:08:41.714 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3769530383,v1:127.0.0.1:6819/3769530383] [v2:127.0.0.1:6820/3769530383,v1:127.0.0.1:6821/3769530383] exists,up 90fffe8d-6606-4437-821c-dccc4185c64e 2026-02-22T22:08:41.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:08:41.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:08:41.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:08:41.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:08:41.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T22:08:41.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:08:41.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:08:41.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T22:08:41.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:08:41.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:08:41.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:08:41.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:08:41.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:08:41.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:08:41.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:08:41.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:08:41.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:08:41.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:08:41.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:08:41.719 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:08:41.719 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:08:41.719 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:08:41.719 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:08:41.719 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:41.719 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:41.720 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:08:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:08:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:08:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:08:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:08:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:08:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:08:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:08:41.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:08:41.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:08:41.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:08:41.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:08:41.721 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:08:41.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5ffc6b56-0e18-45a2-b62a-114dc6058e7b 2026-02-22T22:08:41.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5ffc6b56-0e18-45a2-b62a-114dc6058e7b' 2026-02-22T22:08:41.722 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 5ffc6b56-0e18-45a2-b62a-114dc6058e7b 2026-02-22T22:08:41.722 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:08:41.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpfptpIEKjKxAAAf6sYF3V5M4EBYpTVuqDXQ== 2026-02-22T22:08:41.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpfptpIEKjKxAAAf6sYF3V5M4EBYpTVuqDXQ=="}' 2026-02-22T22:08:41.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5ffc6b56-0e18-45a2-b62a-114dc6058e7b -i td/test-erasure-eio/3/new.json 2026-02-22T22:08:41.933 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:08:41.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T22:08:41.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpfptpIEKjKxAAAf6sYF3V5M4EBYpTVuqDXQ== --osd-uuid 5ffc6b56-0e18-45a2-b62a-114dc6058e7b 2026-02-22T22:08:41.959 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:41.957+0000 7f06eb55d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:41.961 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:41.957+0000 7f06eb55d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:41.962 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:41.957+0000 7f06eb55d8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:41.962 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:41.961+0000 7f06eb55d8c0 -1 bdev(0x55ac24279c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T22:08:41.962 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:41.961+0000 7f06eb55d8c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T22:08:44.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T22:08:44.242 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:08:44.243 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T22:08:44.243 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T22:08:44.243 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:08:44.517 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T22:08:44.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T22:08:44.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:08:44.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:08:44.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:08:44.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:08:44.539 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:44.537+0000 7fbb4ec728c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:44.541 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:44.537+0000 7fbb4ec728c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:44.542 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:44.537+0000 7fbb4ec728c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:44.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T22:08:44.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:08:44.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:08:44.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:08:44.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:08:44.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:44.719 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:08:44.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:08:44.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:44.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:08:44.916 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:45.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:45.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:45.918 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:08:45.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:08:45.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:45.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:08:46.052 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:46.049+0000 7fbb4ec728c0 -1 Falling back to public interface 2026-02-22T22:08:46.121 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:47.123 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:47.123 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:47.123 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:08:47.123 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:08:47.124 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:47.124 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:08:47.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:48.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:48.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:48.319 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:08:48.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:08:48.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:48.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:08:48.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:08:49.070 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:49.065+0000 7fbb4ec728c0 -1 osd.3 0 log_to_monitors true 2026-02-22T22:08:49.516 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:08:49.516 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:08:49.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:08:49.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:08:49.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:08:49.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:08:49.718 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1237154333,v1:127.0.0.1:6827/1237154333] [v2:127.0.0.1:6828/1237154333,v1:127.0.0.1:6829/1237154333] exists,up 5ffc6b56-0e18-45a2-b62a-114dc6058e7b 2026-02-22T22:08:49.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:08:49.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:08:49.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:08:49.720 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T22:08:49.720 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:08:49.720 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:08:49.720 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:08:49.720 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:49.721 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:49.721 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T22:08:49.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T22:08:49.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T22:08:49.782 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:08:49.787 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T22:08:49.790 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:08:20.833+0000 7fa8478eb8c0 0 load: jerasure load: lrc 2026-02-22T22:08:49.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: TEST_rados_get_subread_eio_shard_0: local poolname=pool-jerasure 2026-02-22T22:08:49.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:259: TEST_rados_get_subread_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2026-02-22T22:08:49.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:08:49.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T22:08:49.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-22T22:08:49.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T22:08:49.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-22T22:08:49.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T22:08:49.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-22T22:08:50.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T22:08:50.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T22:08:50.490 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T22:08:50.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:08:51.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T22:08:51.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:08:51.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:08:51.501 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:08:51.502 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:08:51.503 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:08:51.503 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:08:51.503 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:08:51.503 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:08:51.503 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:08:51.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:08:51.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:08:51.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:08:51.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:08:51.613 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:08:51.614 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:08:51.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:08:51.816 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:08:51.816 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:08:51.816 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:08:51.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:08:51.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:08:51.817 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:08:51.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803784 2026-02-22T22:08:51.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803784 2026-02-22T22:08:51.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784' 2026-02-22T22:08:51.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:08:51.893 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:08:51.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=55834574854 2026-02-22T22:08:51.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 55834574854 2026-02-22T22:08:51.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-55834574854' 2026-02-22T22:08:51.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:08:51.969 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:08:52.044 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=77309411332 2026-02-22T22:08:52.044 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 77309411332 2026-02-22T22:08:52.044 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-55834574854 2-77309411332' 2026-02-22T22:08:52.044 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:08:52.044 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:08:52.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=111669149699 2026-02-22T22:08:52.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 111669149699 2026-02-22T22:08:52.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-55834574854 2-77309411332 3-111669149699' 2026-02-22T22:08:52.120 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:08:52.121 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803784 2026-02-22T22:08:52.121 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:08:52.122 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:08:52.123 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803784 2026-02-22T22:08:52.123 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:08:52.124 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803784 2026-02-22T22:08:52.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2026-02-22T22:08:52.125 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803784 2026-02-22T22:08:52.125 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:08:52.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803782 -lt 25769803784 2026-02-22T22:08:52.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:08:53.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:08:53.329 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:08:53.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803784 -lt 25769803784 2026-02-22T22:08:53.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:08:53.528 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-55834574854 2026-02-22T22:08:53.528 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:08:53.529 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:08:53.530 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-55834574854 2026-02-22T22:08:53.530 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:08:53.531 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=55834574854 2026-02-22T22:08:53.532 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 55834574854 2026-02-22T22:08:53.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 55834574854' 2026-02-22T22:08:53.532 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:08:53.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 55834574855 -lt 55834574854 2026-02-22T22:08:53.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:08:53.729 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-77309411332 2026-02-22T22:08:53.729 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:08:53.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:08:53.731 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-77309411332 2026-02-22T22:08:53.731 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:08:53.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=77309411332 2026-02-22T22:08:53.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 77309411332' 2026-02-22T22:08:53.733 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 77309411332 2026-02-22T22:08:53.733 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:08:53.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 77309411333 -lt 77309411332 2026-02-22T22:08:53.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:08:53.945 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-111669149699 2026-02-22T22:08:53.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:08:53.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:08:53.948 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-111669149699 2026-02-22T22:08:53.948 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:08:53.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=111669149699 2026-02-22T22:08:53.949 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 111669149699 2026-02-22T22:08:53.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 111669149699' 2026-02-22T22:08:53.950 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:08:54.153 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 111669149699 -lt 111669149699 2026-02-22T22:08:54.170 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:08:54.170 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:08:54.170 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:08:54.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:08:54.406 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:08:54.406 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:08:54.406 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:08:54.406 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:08:54.406 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:08:54.406 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:08:54.406 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:08:54.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:08:54.597 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:08:54.597 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:08:54.598 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:08:54.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:08:54.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:08:54.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:08:54.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:261: TEST_rados_get_subread_eio_shard_0: local shard_id=0 2026-02-22T22:08:54.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: TEST_rados_get_subread_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 2026-02-22T22:08:54.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2026-02-22T22:08:54.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-22T22:08:54.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-22T22:08:54.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-22T22:08:54.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2026-02-22T22:08:54.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-22T22:08:54.851 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-71671-0 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-71671-0 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-71671-0 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:08:54.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-71671-0 td/test-erasure-eio/ORIGINAL 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-71671-0 td/test-erasure-eio 0 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=obj-eio-71671-0 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=0 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure obj-eio-71671-0 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:08:54.871 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-71671-0 2026-02-22T22:08:54.872 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-71671-0 2026-02-22T22:08:54.872 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:08:55.074 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:08:55.075 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:08:55.075 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:08:55.075 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:08:55.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '1' '2') 2026-02-22T22:08:55.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T22:08:55.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=3 2026-02-22T22:08:55.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T22:08:55.075 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/3/type 2026-02-22T22:08:55.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T22:08:55.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-22T22:08:55.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T22:08:55.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2026-02-22T22:08:55.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T22:08:55.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T22:08:55.077 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T22:08:55.078 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2026-02-22T22:08:55.078 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T22:08:55.078 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T22:08:55.078 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:08:55.078 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:55.078 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:55.079 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T22:08:55.079 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-22T22:08:55.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T22:08:55.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T22:08:55.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T22:08:55.140 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.3 2026-02-22T22:08:55.140 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T22:08:55.140 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T22:08:55.140 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:08:55.140 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:55.140 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:55.140 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T22:08:55.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T22:08:55.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-71671-0 0 2026-02-22T22:08:55.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-71671-0 2026-02-22T22:08:55.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:08:55.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:08:55.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-71671-0 2026-02-22T22:08:55.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:08:55.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:08:55.198 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-71671-0 td/test-erasure-eio/COPY 2026-02-22T22:08:55.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:08:55.218 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:08:55.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2026-02-22T22:08:55.219 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 0 + 1 2026-02-22T22:08:55.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=1 2026-02-22T22:08:55.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-71671-0 td/test-erasure-eio 1 2026-02-22T22:08:55.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T22:08:55.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T22:08:55.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T22:08:55.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T22:08:55.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T22:08:55.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T22:08:55.222 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=obj-eio-71671-0 2026-02-22T22:08:55.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T22:08:55.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T22:08:55.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T22:08:55.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=1 2026-02-22T22:08:55.223 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T22:08:55.223 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure obj-eio-71671-0 2026-02-22T22:08:55.223 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:08:55.223 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-71671-0 2026-02-22T22:08:55.224 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-71671-0 2026-02-22T22:08:55.224 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:08:55.418 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:08:55.419 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:08:55.419 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:08:55.419 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:08:55.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '1' '2') 2026-02-22T22:08:55.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T22:08:55.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=1 2026-02-22T22:08:55.419 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T22:08:55.419 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/1/type 2026-02-22T22:08:55.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T22:08:55.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-22T22:08:55.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T22:08:55.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2026-02-22T22:08:55.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T22:08:55.421 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T22:08:55.421 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T22:08:55.422 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2026-02-22T22:08:55.422 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2026-02-22T22:08:55.422 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2026-02-22T22:08:55.423 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:08:55.423 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:55.423 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:55.423 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.1.asok 2026-02-22T22:08:55.423 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-22T22:08:55.483 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T22:08:55.484 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T22:08:55.484 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T22:08:55.485 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.1 2026-02-22T22:08:55.485 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2026-02-22T22:08:55.485 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2026-02-22T22:08:55.485 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:08:55.486 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:55.486 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:55.486 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.1.asok 2026-02-22T22:08:55.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T22:08:55.486 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-71671-0 1 2026-02-22T22:08:55.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-71671-0 fail 2026-02-22T22:08:55.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:08:55.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:08:55.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-71671-0 2026-02-22T22:08:55.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-22T22:08:55.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-22T22:08:55.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-71671-0 td/test-erasure-eio/COPY 2026-02-22T22:08:55.561 INFO:tasks.workunit.client.0.trial034.stderr:error getting pool-jerasure/obj-eio-71671-0: (5) Input/output error 2026-02-22T22:08:55.563 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-22T22:08:55.563 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2026-02-22T22:08:55.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:263: TEST_rados_get_subread_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2026-02-22T22:08:55.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:08:55.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T22:08:55.806 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T22:08:55.816 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T22:08:56.115 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T22:08:56.124 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T22:08:56.124 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:08:56.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:08:56.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:08:56.125 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:08:56.126 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:08:56.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:08:56.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:08:56.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:08:56.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:08:56.235 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:08:56.235 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:08:56.236 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:08:56.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:08:56.237 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:08:56.237 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:08:56.238 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:08:56.238 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:08:56.239 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:08:56.239 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:08:56.240 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:08:56.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:08:56.241 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:08:56.254 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:08:56.254 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:56.255 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:56.255 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:08:56.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:08:56.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:08:56.256 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T22:08:56.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T22:08:56.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T22:08:56.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T22:08:56.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:08:56.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:08:56.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:08:56.257 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:08:56.257 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:08:56.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:08:56.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:08:56.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:08:56.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:08:56.259 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:08:56.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:08:56.260 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:08:56.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:08:56.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:08:56.262 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:08:56.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:08:56.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:08:56.263 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:08:56.264 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:08:56.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:08:56.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:08:56.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:08:56.267 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:08:56.267 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:56.267 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:56.268 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:08:56.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:08:56.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:08:56.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T22:08:56.270 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:08:56.271 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:56.271 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:56.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T22:08:56.274 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:08:56.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:08:56.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:08:56.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T22:08:56.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T22:08:56.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T22:08:56.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:08:56.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:08:56.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:08:56.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T22:08:56.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T22:08:56.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:08:56.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:08:56.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:08:56.295 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:08:56.295 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:56.295 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:56.295 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:56.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:08:56.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:08:56.326 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:08:56.326 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:08:56.326 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:08:56.326 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:08:56.328 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:08:56.328 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:08:56.328 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:08:56.328 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:08:56.328 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:08:56.328 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:56.328 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:56.328 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:08:56.328 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:08:56.328 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T22:08:56.385 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:08:56.385 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:08:56.385 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:08:56.385 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:08:56.385 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:08:56.386 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:08:56.387 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:08:56.387 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:08:56.387 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:08:56.387 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:56.387 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:56.387 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:08:56.387 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:08:56.387 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T22:08:56.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T22:08:56.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T22:08:56.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:08:56.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:08:56.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:08:56.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T22:08:56.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:08:56.553 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:08:56.554 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:08:56.554 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:08:56.554 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:08:56.554 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:56.554 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:56.555 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:56.555 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:08:56.556 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:08:56.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T22:08:56.573 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:08:56.687 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T22:08:56.695 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:08:57.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T22:08:57.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:08:57.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:08:57.697 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:08:57.697 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:57.697 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:57.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:08:57.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T22:08:57.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T22:08:57.748 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:08:57.754 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T22:08:57.756 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:08:56.312+0000 7fe203255ec0 0 load: jerasure load: lrc 2026-02-22T22:08:57.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_1 td/test-erasure-eio 2026-02-22T22:08:57.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:267: TEST_rados_get_subread_eio_shard_1: local dir=td/test-erasure-eio 2026-02-22T22:08:57.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:268: TEST_rados_get_subread_eio_shard_1: setup_osds 4 2026-02-22T22:08:57.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-22T22:08:57.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T22:08:57.757 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-22T22:08:57.758 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:08:57.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:08:57.760 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:08:57.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:08:57.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:08:57.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:08:57.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T22:08:57.762 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:08:57.763 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5521b24a-199a-439a-83e4-808744f0402b 2026-02-22T22:08:57.763 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5521b24a-199a-439a-83e4-808744f0402b' 2026-02-22T22:08:57.763 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 5521b24a-199a-439a-83e4-808744f0402b 2026-02-22T22:08:57.763 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:08:57.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5fptpb9hILhAATOcsjg13oEA0+Deva6AIbw== 2026-02-22T22:08:57.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5fptpb9hILhAATOcsjg13oEA0+Deva6AIbw=="}' 2026-02-22T22:08:57.776 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5521b24a-199a-439a-83e4-808744f0402b -i td/test-erasure-eio/0/new.json 2026-02-22T22:08:57.886 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:08:57.894 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T22:08:57.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5fptpb9hILhAATOcsjg13oEA0+Deva6AIbw== --osd-uuid 5521b24a-199a-439a-83e4-808744f0402b 2026-02-22T22:08:57.911 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:57.908+0000 7fe159df58c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:57.912 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:57.908+0000 7fe159df58c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:57.913 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:57.908+0000 7fe159df58c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:08:57.913 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:57.908+0000 7fe159df58c0 -1 bdev(0x55964ced9c00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:08:57.914 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:08:57.908+0000 7fe159df58c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T22:09:00.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T22:09:00.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:09:00.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:09:00.462 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T22:09:00.462 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:09:00.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:09:00.727 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T22:09:00.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:09:00.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:09:00.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:09:00.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:09:00.743 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:00.740+0000 7fc28437c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:00.744 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:00.740+0000 7fc28437c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:00.745 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:00.740+0000 7fc28437c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:00.928 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:09:00.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:09:00.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:09:00.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:09:00.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:09:00.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:00.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:09:00.930 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:09:00.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:00.930 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:09:01.125 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:02.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:02.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:02.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:09:02.127 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:09:02.127 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:02.129 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:09:02.252 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:02.248+0000 7fc28437c8c0 -1 Falling back to public interface 2026-02-22T22:09:02.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:03.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:03.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:03.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:09:03.320 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:09:03.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:03.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:09:03.515 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:04.516 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:04.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:04.517 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:09:04.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:09:04.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:04.517 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:09:04.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:05.306 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:05.300+0000 7fc28437c8c0 -1 osd.0 0 log_to_monitors true 2026-02-22T22:09:05.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:05.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:05.733 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:09:05.734 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:09:05.734 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:05.734 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:09:05.944 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1929875361,v1:127.0.0.1:6803/1929875361] [v2:127.0.0.1:6804/1929875361,v1:127.0.0.1:6805/1929875361] exists,up 5521b24a-199a-439a-83e4-808744f0402b 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:09:05.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:09:05.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:09:05.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:09:05.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:09:05.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:09:05.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:09:05.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:09:05.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:09:05.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:09:05.948 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:09:05.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=649c621a-65a0-451f-aff1-c8303dacc4eb 2026-02-22T22:09:05.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 649c621a-65a0-451f-aff1-c8303dacc4eb' 2026-02-22T22:09:05.949 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 649c621a-65a0-451f-aff1-c8303dacc4eb 2026-02-22T22:09:05.949 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:09:05.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBfptpAqRQORAAv8uyPNlWr+D0KWynhbvUBQ== 2026-02-22T22:09:05.961 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBfptpAqRQORAAv8uyPNlWr+D0KWynhbvUBQ=="}' 2026-02-22T22:09:05.961 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 649c621a-65a0-451f-aff1-c8303dacc4eb -i td/test-erasure-eio/1/new.json 2026-02-22T22:09:06.170 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:09:06.181 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T22:09:06.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBfptpAqRQORAAv8uyPNlWr+D0KWynhbvUBQ== --osd-uuid 649c621a-65a0-451f-aff1-c8303dacc4eb 2026-02-22T22:09:06.204 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:06.200+0000 7f137718b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:06.205 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:06.200+0000 7f137718b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:06.207 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:06.204+0000 7f137718b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:06.207 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:06.204+0000 7f137718b8c0 -1 bdev(0x555b2f0ddc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:09:06.207 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:06.204+0000 7f137718b8c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T22:09:08.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T22:09:08.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:09:08.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:09:08.469 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T22:09:08.469 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:09:08.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:09:08.739 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T22:09:08.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:09:08.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:09:08.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:09:08.741 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:09:08.761 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:08.756+0000 7f1fdec528c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:08.763 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:08.760+0000 7f1fdec528c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:08.764 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:08.760+0000 7f1fdec528c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:08.937 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:09:08.938 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:09:08.938 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:09:08.938 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:09:08.938 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:09:08.938 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:08.939 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:09:08.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:09:08.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:08.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:09:09.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:10.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:10.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:10.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:09:10.135 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:09:10.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:10.135 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:09:10.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:10.484 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:10.480+0000 7f1fdec528c0 -1 Falling back to public interface 2026-02-22T22:09:11.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:11.337 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:11.338 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:09:11.338 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:09:11.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:11.339 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:09:11.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:12.534 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:12.534 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:12.535 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:09:12.535 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:09:12.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:12.536 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:09:12.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:13.458 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:13.452+0000 7f1fdec528c0 -1 osd.1 0 log_to_monitors true 2026-02-22T22:09:13.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:13.737 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:13.738 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:09:13.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:09:13.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:13.738 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:09:13.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:14.941 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:14.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:14.942 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:09:14.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:09:14.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:14.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:09:15.141 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3677855941,v1:127.0.0.1:6811/3677855941] [v2:127.0.0.1:6812/3677855941,v1:127.0.0.1:6813/3677855941] exists,up 649c621a-65a0-451f-aff1-c8303dacc4eb 2026-02-22T22:09:15.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:09:15.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:09:15.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:09:15.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:09:15.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T22:09:15.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:09:15.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:09:15.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:09:15.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:09:15.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:09:15.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:09:15.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:09:15.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:09:15.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:09:15.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:09:15.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:09:15.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:09:15.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:09:15.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:09:15.147 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:09:15.147 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:09:15.147 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:09:15.147 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:09:15.147 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:15.147 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:15.147 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:15.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:15.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:09:15.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:09:15.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:09:15.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:09:15.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:09:15.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:09:15.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:09:15.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:09:15.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:09:15.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:09:15.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:09:15.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:09:15.148 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:09:15.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9ef7f014-c732-403e-8229-91f70a5e4287 2026-02-22T22:09:15.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9ef7f014-c732-403e-8229-91f70a5e4287' 2026-02-22T22:09:15.149 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 9ef7f014-c732-403e-8229-91f70a5e4287 2026-02-22T22:09:15.150 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:09:15.158 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLfptp/AR2CRAAm64Pl5udGRf0sE8Z+PHOEA== 2026-02-22T22:09:15.158 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLfptp/AR2CRAAm64Pl5udGRf0sE8Z+PHOEA=="}' 2026-02-22T22:09:15.158 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9ef7f014-c732-403e-8229-91f70a5e4287 -i td/test-erasure-eio/2/new.json 2026-02-22T22:09:15.369 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:09:15.378 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T22:09:15.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLfptp/AR2CRAAm64Pl5udGRf0sE8Z+PHOEA== --osd-uuid 9ef7f014-c732-403e-8229-91f70a5e4287 2026-02-22T22:09:15.396 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:15.392+0000 7f7181a468c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:15.397 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:15.392+0000 7f7181a468c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:15.398 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:15.392+0000 7f7181a468c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:15.399 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:15.396+0000 7f7181a468c0 -1 bdev(0x5639b91b7c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:09:15.399 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:15.396+0000 7f7181a468c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T22:09:17.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T22:09:17.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:09:17.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:09:17.677 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T22:09:17.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:09:17.948 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T22:09:17.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:09:17.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:09:17.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:09:17.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:09:17.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:09:17.971 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:17.968+0000 7fe1b33ff8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:17.973 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:17.968+0000 7fe1b33ff8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:17.974 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:17.968+0000 7fe1b33ff8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:18.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:09:18.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:09:18.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:09:18.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:09:18.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:09:18.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:18.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:09:18.140 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:09:18.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:18.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:09:18.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:19.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:19.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:19.346 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:09:19.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:09:19.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:19.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:09:19.524 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:19.520+0000 7fe1b33ff8c0 -1 Falling back to public interface 2026-02-22T22:09:19.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:20.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:20.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:20.549 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:09:20.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:09:20.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:20.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:09:20.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:21.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:21.751 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:21.751 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:09:21.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:09:21.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:21.752 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:09:21.947 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:22.562 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:22.560+0000 7fe1b33ff8c0 -1 osd.2 0 log_to_monitors true 2026-02-22T22:09:22.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:22.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:22.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:09:22.949 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:09:22.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:22.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:09:23.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:24.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:24.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:24.163 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:09:24.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:09:24.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:24.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:09:24.360 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3681839412,v1:127.0.0.1:6819/3681839412] [v2:127.0.0.1:6820/3681839412,v1:127.0.0.1:6821/3681839412] exists,up 9ef7f014-c732-403e-8229-91f70a5e4287 2026-02-22T22:09:24.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:09:24.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:09:24.360 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:24.361 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:09:24.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:09:24.364 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:09:24.365 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ac51b584-225d-4de2-aa5f-4848d52d7e93 2026-02-22T22:09:24.365 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 ac51b584-225d-4de2-aa5f-4848d52d7e93' 2026-02-22T22:09:24.365 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 ac51b584-225d-4de2-aa5f-4848d52d7e93 2026-02-22T22:09:24.365 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:09:24.377 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCUfptpcs6LFhAAMCec3xnnKmquQKYD2IsrtA== 2026-02-22T22:09:24.377 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCUfptpcs6LFhAAMCec3xnnKmquQKYD2IsrtA=="}' 2026-02-22T22:09:24.377 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ac51b584-225d-4de2-aa5f-4848d52d7e93 -i td/test-erasure-eio/3/new.json 2026-02-22T22:09:24.584 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:09:24.592 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T22:09:24.593 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCUfptpcs6LFhAAMCec3xnnKmquQKYD2IsrtA== --osd-uuid ac51b584-225d-4de2-aa5f-4848d52d7e93 2026-02-22T22:09:24.608 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:24.608+0000 7f52d2c1e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:24.610 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:24.608+0000 7f52d2c1e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:24.611 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:24.612+0000 7f52d2c1e8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:24.611 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:24.612+0000 7f52d2c1e8c0 -1 bdev(0x55e68b181c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T22:09:24.611 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:24.612+0000 7f52d2c1e8c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T22:09:26.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T22:09:26.895 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:09:26.896 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T22:09:26.897 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T22:09:26.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:09:27.165 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T22:09:27.165 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T22:09:27.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:09:27.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:09:27.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:09:27.166 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:09:27.181 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:27.180+0000 7fc31bbee8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:27.182 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:27.180+0000 7fc31bbee8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:27.183 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:27.184+0000 7fc31bbee8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:27.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T22:09:27.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:09:27.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:09:27.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:09:27.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:09:27.362 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:09:27.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:27.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:09:27.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:27.363 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:09:27.563 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:28.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:28.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:28.565 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:09:28.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:09:28.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:28.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:09:28.672 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:28.672+0000 7fc31bbee8c0 -1 Falling back to public interface 2026-02-22T22:09:28.763 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:29.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:29.765 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:29.765 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:09:29.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:09:29.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:29.766 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:09:29.961 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:30.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:30.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:30.963 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:09:30.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:09:30.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:30.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:09:31.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:31.894 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:31.892+0000 7fc31bbee8c0 -1 osd.3 0 log_to_monitors true 2026-02-22T22:09:32.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:32.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:32.164 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:09:32.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:09:32.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:32.164 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:09:32.364 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:33.366 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:33.366 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:33.366 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:09:33.367 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:09:33.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:33.367 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:09:33.564 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1715718731,v1:127.0.0.1:6827/1715718731] [v2:127.0.0.1:6828/1715718731,v1:127.0.0.1:6829/1715718731] exists,up ac51b584-225d-4de2-aa5f-4848d52d7e93 2026-02-22T22:09:33.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:09:33.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:09:33.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:09:33.565 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T22:09:33.565 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:09:33.565 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:09:33.566 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:09:33.566 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:33.566 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:33.566 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T22:09:33.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T22:09:33.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T22:09:33.619 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:09:33.625 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T22:09:33.627 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:09:02.252+0000 7fc28437c8c0 0 load: jerasure load: lrc 2026-02-22T22:09:33.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:270: TEST_rados_get_subread_eio_shard_1: local poolname=pool-jerasure 2026-02-22T22:09:33.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:271: TEST_rados_get_subread_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2026-02-22T22:09:33.628 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:09:33.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T22:09:33.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-22T22:09:33.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T22:09:33.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-22T22:09:33.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T22:09:33.629 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-22T22:09:33.885 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T22:09:33.886 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T22:09:34.241 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T22:09:34.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:09:35.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T22:09:35.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:09:35.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:09:35.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:09:35.254 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:09:35.254 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:09:35.254 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:09:35.254 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:09:35.255 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:09:35.255 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:09:35.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:09:35.361 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:09:35.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:09:35.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:09:35.362 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:09:35.362 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:09:35.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:09:35.567 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:09:35.567 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:09:35.567 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:09:35.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:09:35.567 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:09:35.567 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:09:35.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803784 2026-02-22T22:09:35.643 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803784 2026-02-22T22:09:35.643 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784' 2026-02-22T22:09:35.643 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:09:35.643 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:09:35.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607559 2026-02-22T22:09:35.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607559 2026-02-22T22:09:35.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-51539607559' 2026-02-22T22:09:35.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:09:35.718 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:09:35.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=73014444037 2026-02-22T22:09:35.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 73014444037 2026-02-22T22:09:35.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-51539607559 2-73014444037' 2026-02-22T22:09:35.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:09:35.792 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:09:35.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182404 2026-02-22T22:09:35.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182404 2026-02-22T22:09:35.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-51539607559 2-73014444037 3-107374182404' 2026-02-22T22:09:35.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:09:35.868 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803784 2026-02-22T22:09:35.868 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:09:35.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:09:35.870 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803784 2026-02-22T22:09:35.870 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:09:35.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803784 2026-02-22T22:09:35.871 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803784 2026-02-22T22:09:35.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2026-02-22T22:09:35.872 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:09:36.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803782 -lt 25769803784 2026-02-22T22:09:36.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:09:37.079 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:09:37.079 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:09:37.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803785 -lt 25769803784 2026-02-22T22:09:37.283 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:09:37.284 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607559 2026-02-22T22:09:37.284 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:09:37.285 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:09:37.285 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607559 2026-02-22T22:09:37.286 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:09:37.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607559 2026-02-22T22:09:37.287 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607559 2026-02-22T22:09:37.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607559' 2026-02-22T22:09:37.288 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:09:37.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607559 -lt 51539607559 2026-02-22T22:09:37.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:09:37.481 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-73014444037 2026-02-22T22:09:37.481 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:09:37.482 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:09:37.483 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-73014444037 2026-02-22T22:09:37.483 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:09:37.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=73014444037 2026-02-22T22:09:37.485 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 73014444037 2026-02-22T22:09:37.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 73014444037' 2026-02-22T22:09:37.485 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:09:37.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 73014444037 -lt 73014444037 2026-02-22T22:09:37.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:09:37.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182404 2026-02-22T22:09:37.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:09:37.698 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:09:37.699 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182404 2026-02-22T22:09:37.699 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:09:37.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182404 2026-02-22T22:09:37.701 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 107374182404 2026-02-22T22:09:37.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182404' 2026-02-22T22:09:37.701 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:09:37.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182404 -lt 107374182404 2026-02-22T22:09:37.905 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:09:37.906 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:09:37.906 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:09:38.158 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:09:38.158 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:09:38.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:09:38.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:09:38.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:09:38.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:09:38.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:09:38.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:09:38.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:09:38.349 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:09:38.349 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:09:38.349 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:09:38.601 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:09:38.601 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:09:38.601 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:09:38.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:273: TEST_rados_get_subread_eio_shard_1: local shard_id=1 2026-02-22T22:09:38.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:274: TEST_rados_get_subread_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 2026-02-22T22:09:38.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2026-02-22T22:09:38.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-22T22:09:38.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-22T22:09:38.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-22T22:09:38.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2026-02-22T22:09:38.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-22T22:09:38.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2026-02-22T22:09:38.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-22T22:09:38.602 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-71671-1 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-71671-1 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-71671-1 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:09:38.603 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-71671-1 td/test-erasure-eio/ORIGINAL 2026-02-22T22:09:38.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-71671-1 td/test-erasure-eio 1 2026-02-22T22:09:38.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T22:09:38.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T22:09:38.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T22:09:38.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T22:09:38.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T22:09:38.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T22:09:38.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=obj-eio-71671-1 2026-02-22T22:09:38.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T22:09:38.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T22:09:38.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T22:09:38.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=1 2026-02-22T22:09:38.621 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T22:09:38.621 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure obj-eio-71671-1 2026-02-22T22:09:38.621 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:09:38.621 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-71671-1 2026-02-22T22:09:38.621 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-71671-1 2026-02-22T22:09:38.621 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:09:38.820 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:09:38.821 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:09:38.821 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:09:38.821 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:09:38.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '1' '2') 2026-02-22T22:09:38.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T22:09:38.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=1 2026-02-22T22:09:38.821 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T22:09:38.821 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/1/type 2026-02-22T22:09:38.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T22:09:38.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-22T22:09:38.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T22:09:38.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2026-02-22T22:09:38.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T22:09:38.823 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T22:09:38.823 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T22:09:38.823 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2026-02-22T22:09:38.823 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2026-02-22T22:09:38.823 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2026-02-22T22:09:38.824 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:09:38.824 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:38.825 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:38.825 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.1.asok 2026-02-22T22:09:38.825 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-22T22:09:38.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T22:09:38.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T22:09:38.885 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T22:09:38.885 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.1 2026-02-22T22:09:38.885 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2026-02-22T22:09:38.885 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2026-02-22T22:09:38.885 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:09:38.885 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:38.885 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:38.885 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.1.asok 2026-02-22T22:09:38.885 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T22:09:38.886 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-71671-1 1 2026-02-22T22:09:38.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-71671-1 2026-02-22T22:09:38.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:09:38.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:09:38.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-71671-1 2026-02-22T22:09:38.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:09:38.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:09:38.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-71671-1 td/test-erasure-eio/COPY 2026-02-22T22:09:38.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:09:38.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:09:38.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2026-02-22T22:09:38.964 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2026-02-22T22:09:38.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=2 2026-02-22T22:09:38.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-71671-1 td/test-erasure-eio 2 2026-02-22T22:09:38.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T22:09:38.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T22:09:38.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T22:09:38.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T22:09:38.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T22:09:38.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T22:09:38.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=obj-eio-71671-1 2026-02-22T22:09:38.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T22:09:38.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T22:09:38.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T22:09:38.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=2 2026-02-22T22:09:38.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T22:09:38.968 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure obj-eio-71671-1 2026-02-22T22:09:38.968 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:09:38.968 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-71671-1 2026-02-22T22:09:38.969 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-71671-1 2026-02-22T22:09:38.969 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:09:39.160 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:09:39.161 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:09:39.161 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:09:39.161 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:09:39.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '1' '2') 2026-02-22T22:09:39.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T22:09:39.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=2 2026-02-22T22:09:39.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T22:09:39.161 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/2/type 2026-02-22T22:09:39.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T22:09:39.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2026-02-22T22:09:39.162 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T22:09:39.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2026-02-22T22:09:39.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T22:09:39.163 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T22:09:39.163 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T22:09:39.164 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2026-02-22T22:09:39.164 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2026-02-22T22:09:39.164 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2026-02-22T22:09:39.164 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:09:39.164 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:39.165 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:39.165 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.2.asok 2026-02-22T22:09:39.165 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2026-02-22T22:09:39.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T22:09:39.225 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T22:09:39.226 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T22:09:39.226 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.2 2026-02-22T22:09:39.226 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2026-02-22T22:09:39.226 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2026-02-22T22:09:39.227 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:09:39.227 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:39.228 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:39.228 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.2.asok 2026-02-22T22:09:39.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T22:09:39.228 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.2.asok injectdataerr pool-jerasure obj-eio-71671-1 2 2026-02-22T22:09:39.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-71671-1 fail 2026-02-22T22:09:39.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:09:39.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:09:39.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-71671-1 2026-02-22T22:09:39.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-22T22:09:39.286 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-22T22:09:39.287 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-71671-1 td/test-erasure-eio/COPY 2026-02-22T22:09:39.303 INFO:tasks.workunit.client.0.trial034.stderr:error getting pool-jerasure/obj-eio-71671-1: (5) Input/output error 2026-02-22T22:09:39.304 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-22T22:09:39.304 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2026-02-22T22:09:39.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:275: TEST_rados_get_subread_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2026-02-22T22:09:39.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:09:39.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T22:09:39.544 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T22:09:39.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T22:09:39.813 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T22:09:39.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T22:09:39.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:09:39.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:09:39.825 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:09:39.825 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:09:39.825 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:09:39.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:09:39.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:09:39.826 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:09:39.935 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:09:39.936 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:09:39.936 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:09:39.937 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:09:39.938 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:09:39.938 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:09:39.938 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:09:39.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:09:39.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:09:39.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:09:39.940 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:09:39.941 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:09:39.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:09:39.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:09:39.953 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:09:39.954 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:39.954 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:39.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:09:39.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:09:39.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:09:39.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T22:09:39.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T22:09:39.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T22:09:39.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T22:09:39.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:09:39.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:09:39.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:09:39.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:09:39.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:09:39.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:09:39.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:09:39.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:09:39.959 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:09:39.959 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:09:39.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:09:39.961 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:09:39.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:09:39.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:09:39.962 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:09:39.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:09:39.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:09:39.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:09:39.964 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:09:39.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:09:39.966 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:09:39.967 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:09:39.968 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:09:39.968 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:39.968 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:39.968 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:09:39.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:09:39.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:09:39.969 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T22:09:39.971 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:09:39.971 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:39.971 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:39.971 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T22:09:39.973 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:09:39.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:09:39.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:09:39.973 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T22:09:39.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T22:09:39.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T22:09:39.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:09:39.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:09:39.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:09:39.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T22:09:39.974 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T22:09:39.995 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:09:39.995 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:09:39.995 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:09:39.996 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:09:39.996 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:39.996 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:39.996 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:39.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:09:40.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:09:40.023 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:09:40.024 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:09:40.024 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:09:40.024 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:09:40.025 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:09:40.025 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:09:40.025 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:09:40.025 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:09:40.026 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:09:40.026 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:40.026 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:40.026 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:09:40.026 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:09:40.026 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T22:09:40.081 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:09:40.081 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:09:40.081 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:09:40.082 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:09:40.082 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:09:40.083 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:09:40.083 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:09:40.083 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:09:40.083 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:09:40.083 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:40.084 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:40.084 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:09:40.084 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:09:40.084 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T22:09:40.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T22:09:40.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T22:09:40.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:09:40.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:09:40.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:09:40.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T22:09:40.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:09:40.251 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:09:40.251 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:09:40.251 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:09:40.251 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:09:40.251 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:40.251 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:40.251 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:40.252 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:09:40.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:09:40.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T22:09:40.270 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:09:40.375 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T22:09:40.383 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:09:41.386 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T22:09:41.386 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:09:41.386 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:09:41.386 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:09:41.387 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:41.387 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:41.387 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:09:41.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T22:09:41.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T22:09:41.434 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:09:41.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T22:09:41.442 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:09:40.011+0000 7fec4be62ec0 0 load: jerasure load: lrc 2026-02-22T22:09:41.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_missing td/test-erasure-eio 2026-02-22T22:09:41.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:282: TEST_rados_get_subread_missing: local dir=td/test-erasure-eio 2026-02-22T22:09:41.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:283: TEST_rados_get_subread_missing: setup_osds 4 2026-02-22T22:09:41.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-22T22:09:41.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T22:09:41.443 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-22T22:09:41.447 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-22T22:09:41.447 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:09:41.447 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T22:09:41.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:09:41.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:09:41.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:09:41.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:09:41.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T22:09:41.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:09:41.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:09:41.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:09:41.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:09:41.448 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T22:09:41.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T22:09:41.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:09:41.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:09:41.449 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:09:41.449 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:09:41.449 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:09:41.449 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:09:41.449 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:09:41.449 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:41.449 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:41.449 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:41.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:41.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:09:41.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:09:41.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:09:41.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:09:41.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:09:41.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:09:41.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:09:41.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:09:41.450 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:09:41.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:09:41.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:09:41.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T22:09:41.451 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:09:41.451 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dad091c6-bbc8-423e-a6fa-43a74c45f06e 2026-02-22T22:09:41.452 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dad091c6-bbc8-423e-a6fa-43a74c45f06e' 2026-02-22T22:09:41.452 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 dad091c6-bbc8-423e-a6fa-43a74c45f06e 2026-02-22T22:09:41.452 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:09:41.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQClfptpj+hzGxAAWTD4MN7LJp1MCjRSjTllLw== 2026-02-22T22:09:41.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQClfptpj+hzGxAAWTD4MN7LJp1MCjRSjTllLw=="}' 2026-02-22T22:09:41.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dad091c6-bbc8-423e-a6fa-43a74c45f06e -i td/test-erasure-eio/0/new.json 2026-02-22T22:09:41.568 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:09:41.575 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T22:09:41.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQClfptpj+hzGxAAWTD4MN7LJp1MCjRSjTllLw== --osd-uuid dad091c6-bbc8-423e-a6fa-43a74c45f06e 2026-02-22T22:09:41.592 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:41.591+0000 7fb4f286a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:41.593 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:41.591+0000 7fb4f286a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:41.594 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:41.591+0000 7fb4f286a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:41.595 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:41.595+0000 7fb4f286a8c0 -1 bdev(0x563e7581dc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:09:41.595 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:41.595+0000 7fb4f286a8c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T22:09:44.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T22:09:44.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:09:44.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:09:44.206 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T22:09:44.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:09:44.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:09:44.478 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T22:09:44.479 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:09:44.479 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:09:44.479 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:09:44.479 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:09:44.501 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:44.499+0000 7fa2e54118c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:44.503 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:44.503+0000 7fa2e54118c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:44.504 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:44.503+0000 7fa2e54118c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:44.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:09:44.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:09:44.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:09:44.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:09:44.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:09:44.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:44.675 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:09:44.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:09:44.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:44.676 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:09:44.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:45.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:45.876 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:45.876 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:09:45.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:09:45.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:45.877 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:09:45.976 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:45.975+0000 7fa2e54118c0 -1 Falling back to public interface 2026-02-22T22:09:46.078 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:47.079 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:47.079 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:47.080 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:09:47.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:09:47.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:47.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:09:47.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:48.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:48.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:48.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:09:48.278 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:09:48.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:48.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:09:48.479 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:49.224 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:49.223+0000 7fa2e54118c0 -1 osd.0 0 log_to_monitors true 2026-02-22T22:09:49.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:49.480 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:49.481 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:09:49.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:09:49.482 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:49.482 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:09:49.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:50.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:50.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:50.674 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:09:50.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:09:50.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:50.675 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:09:50.876 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3861006214,v1:127.0.0.1:6803/3861006214] [v2:127.0.0.1:6804/3861006214,v1:127.0.0.1:6805/3861006214] exists,up dad091c6-bbc8-423e-a6fa-43a74c45f06e 2026-02-22T22:09:50.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:09:50.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:09:50.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:09:50.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:09:50.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T22:09:50.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:09:50.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:09:50.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:09:50.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:09:50.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:09:50.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:09:50.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:09:50.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:09:50.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:09:50.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:09:50.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:09:50.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:09:50.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:09:50.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:09:50.881 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:09:50.882 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:09:50.882 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:09:50.882 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:09:50.882 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:50.882 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:50.882 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:50.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:50.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:09:50.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:09:50.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:09:50.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:09:50.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:09:50.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:09:50.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:09:50.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:09:50.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:09:50.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:09:50.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:09:50.883 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:09:50.883 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:09:50.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0eafe266-26e2-4742-885d-3020c8bd1297 2026-02-22T22:09:50.884 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0eafe266-26e2-4742-885d-3020c8bd1297' 2026-02-22T22:09:50.884 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 0eafe266-26e2-4742-885d-3020c8bd1297 2026-02-22T22:09:50.885 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:09:50.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCufptpS8tDNRAAlo19i9ApbifsxUWMQZpNZQ== 2026-02-22T22:09:50.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCufptpS8tDNRAAlo19i9ApbifsxUWMQZpNZQ=="}' 2026-02-22T22:09:50.893 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0eafe266-26e2-4742-885d-3020c8bd1297 -i td/test-erasure-eio/1/new.json 2026-02-22T22:09:51.100 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:09:51.109 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T22:09:51.110 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCufptpS8tDNRAAlo19i9ApbifsxUWMQZpNZQ== --osd-uuid 0eafe266-26e2-4742-885d-3020c8bd1297 2026-02-22T22:09:51.126 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:51.123+0000 7ff3508f48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:51.128 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:51.127+0000 7ff3508f48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:51.129 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:51.127+0000 7ff3508f48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:51.129 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:51.127+0000 7ff3508f48c0 -1 bdev(0x55fb00c2fc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:09:51.129 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:51.127+0000 7ff3508f48c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T22:09:53.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T22:09:53.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:09:53.464 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:09:53.465 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T22:09:53.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:09:53.727 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T22:09:53.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:09:53.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:09:53.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:09:53.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:09:53.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:09:53.750 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:53.747+0000 7f821b4e48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:53.751 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:53.751+0000 7f821b4e48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:53.752 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:53.751+0000 7f821b4e48c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:53.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:09:53.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:09:53.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:09:53.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:09:53.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:09:53.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:53.920 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:09:53.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:09:53.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:53.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:09:54.116 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:55.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:55.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:55.118 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:09:55.119 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:09:55.119 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:55.119 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:09:55.256 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:55.255+0000 7f821b4e48c0 -1 Falling back to public interface 2026-02-22T22:09:55.318 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:56.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:56.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:56.319 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:09:56.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:09:56.319 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:56.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:09:56.516 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:57.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:57.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:57.518 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:09:57.519 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:09:57.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:57.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:09:57.716 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:09:58.257 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:58.255+0000 7f821b4e48c0 -1 osd.1 0 log_to_monitors true 2026-02-22T22:09:58.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:09:58.717 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:09:58.718 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:09:58.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:09:58.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:09:58.719 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:09:58.911 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1184418675,v1:127.0.0.1:6811/1184418675] [v2:127.0.0.1:6812/1184418675,v1:127.0.0.1:6813/1184418675] exists,up 0eafe266-26e2-4742-885d-3020c8bd1297 2026-02-22T22:09:58.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:09:58.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:09:58.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:09:58.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:09:58.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T22:09:58.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:09:58.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:09:58.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:09:58.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:09:58.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:09:58.918 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:09:58.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:09:58.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:09:58.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:09:58.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:09:58.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:09:58.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:09:58.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:09:58.919 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:09:58.920 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:09:58.920 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:09:58.920 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:09:58.920 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:09:58.920 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:09:58.920 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:09:58.920 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:58.920 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:09:58.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:09:58.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:09:58.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:09:58.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:09:58.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:09:58.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:09:58.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:09:58.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:09:58.921 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:09:58.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:09:58.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:09:58.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:09:58.922 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:09:58.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a28e3b93-ac60-4cab-a420-8bd96b96c2dd 2026-02-22T22:09:58.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a28e3b93-ac60-4cab-a420-8bd96b96c2dd' 2026-02-22T22:09:58.922 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 a28e3b93-ac60-4cab-a420-8bd96b96c2dd 2026-02-22T22:09:58.923 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:09:58.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2fptpHPJeNxAAvIeMZb63DyUt1BaBMTZ8yw== 2026-02-22T22:09:58.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2fptpHPJeNxAAvIeMZb63DyUt1BaBMTZ8yw=="}' 2026-02-22T22:09:58.929 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a28e3b93-ac60-4cab-a420-8bd96b96c2dd -i td/test-erasure-eio/2/new.json 2026-02-22T22:09:59.140 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:09:59.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T22:09:59.150 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2fptpHPJeNxAAvIeMZb63DyUt1BaBMTZ8yw== --osd-uuid a28e3b93-ac60-4cab-a420-8bd96b96c2dd 2026-02-22T22:09:59.165 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:59.163+0000 7fd8938c38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:59.166 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:59.167+0000 7fd8938c38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:59.168 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:59.167+0000 7fd8938c38c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:09:59.168 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:59.167+0000 7fd8938c38c0 -1 bdev(0x56188efebc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:09:59.168 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:09:59.167+0000 7fd8938c38c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T22:10:01.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T22:10:01.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:10:01.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:10:01.437 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T22:10:01.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:10:01.709 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:10:01.710 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T22:10:01.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:10:01.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:10:01.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:10:01.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:10:01.732 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:01.731+0000 7f32dd5968c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:01.733 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:01.731+0000 7f32dd5968c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:01.734 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:01.731+0000 7f32dd5968c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:01.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:10:01.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:10:01.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:10:01.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:10:01.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:10:01.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:01.908 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:10:01.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:10:01.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:01.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:10:02.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:03.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:03.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:03.106 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:10:03.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:10:03.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:03.106 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:10:03.276 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:03.275+0000 7f32dd5968c0 -1 Falling back to public interface 2026-02-22T22:10:03.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:04.307 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:04.307 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:04.307 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:10:04.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:10:04.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:04.308 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:10:04.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:05.505 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:05.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:05.506 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:10:05.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:10:05.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:05.507 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:10:05.705 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:06.275 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:06.275+0000 7f32dd5968c0 -1 osd.2 0 log_to_monitors true 2026-02-22T22:10:06.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:06.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:06.707 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:10:06.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:10:06.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:06.708 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:10:06.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:07.906 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:07.907 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:07.907 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:10:07.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:10:07.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:07.908 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:10:08.100 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/438997238,v1:127.0.0.1:6819/438997238] [v2:127.0.0.1:6820/438997238,v1:127.0.0.1:6821/438997238] exists,up a28e3b93-ac60-4cab-a420-8bd96b96c2dd 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:08.101 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:08.102 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:10:08.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:10:08.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:10:08.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:10:08.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:10:08.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:10:08.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:10:08.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:10:08.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:10:08.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:10:08.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:10:08.102 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:10:08.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:10:08.103 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:10:08.104 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:10:08.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43017b86-a1ea-466b-aeb1-a995df069871 2026-02-22T22:10:08.105 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 43017b86-a1ea-466b-aeb1-a995df069871 2026-02-22T22:10:08.105 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 43017b86-a1ea-466b-aeb1-a995df069871' 2026-02-22T22:10:08.105 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:10:08.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDAfptp9WALBxAAo9noC4eEExkx+QxHShC0Vw== 2026-02-22T22:10:08.117 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDAfptp9WALBxAAo9noC4eEExkx+QxHShC0Vw=="}' 2026-02-22T22:10:08.118 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43017b86-a1ea-466b-aeb1-a995df069871 -i td/test-erasure-eio/3/new.json 2026-02-22T22:10:08.321 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:10:08.330 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T22:10:08.331 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDAfptp9WALBxAAo9noC4eEExkx+QxHShC0Vw== --osd-uuid 43017b86-a1ea-466b-aeb1-a995df069871 2026-02-22T22:10:08.347 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:08.347+0000 7fc075b068c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:08.348 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:08.347+0000 7fc075b068c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:08.349 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:08.347+0000 7fc075b068c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:08.350 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:08.347+0000 7fc075b068c0 -1 bdev(0x5592d30a9c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T22:10:08.350 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:08.347+0000 7fc075b068c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T22:10:10.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T22:10:10.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:10:10.608 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T22:10:10.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T22:10:10.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:10:10.871 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T22:10:10.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T22:10:10.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:10:10.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:10:10.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:10:10.873 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:10:10.895 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:10.895+0000 7f157e4598c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:10.896 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:10.895+0000 7f157e4598c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:10.898 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:10.895+0000 7f157e4598c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:11.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T22:10:11.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:10:11.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:10:11.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:10:11.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:10:11.075 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:10:11.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:11.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:10:11.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:11.076 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:10:11.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:12.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:12.274 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:12.274 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:10:12.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:10:12.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:12.275 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:10:12.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:12.620 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:12.619+0000 7f157e4598c0 -1 Falling back to public interface 2026-02-22T22:10:13.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:13.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:13.478 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:10:13.478 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:10:13.479 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:13.479 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:10:13.669 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:14.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:14.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:14.670 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:10:14.670 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:10:14.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:14.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:10:14.864 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:15.611 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:15.611+0000 7f157e4598c0 -1 osd.3 0 log_to_monitors true 2026-02-22T22:10:15.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:15.866 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:15.866 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:10:15.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:10:15.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:15.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:10:16.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:17.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:17.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:17.066 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:10:17.067 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:10:17.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:17.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:10:17.256 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 25 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3734027043,v1:127.0.0.1:6827/3734027043] [v2:127.0.0.1:6828/3734027043,v1:127.0.0.1:6829/3734027043] exists,up 43017b86-a1ea-466b-aeb1-a995df069871 2026-02-22T22:10:17.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:10:17.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:10:17.257 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:10:17.258 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T22:10:17.258 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:10:17.258 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:10:17.258 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:10:17.258 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:17.258 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:17.259 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T22:10:17.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T22:10:17.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T22:10:17.320 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:10:17.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T22:10:17.327 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:09:45.979+0000 7fa2e54118c0 0 load: jerasure load: lrc 2026-02-22T22:10:17.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:285: TEST_rados_get_subread_missing: local poolname=pool-jerasure 2026-02-22T22:10:17.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:286: TEST_rados_get_subread_missing: create_erasure_coded_pool pool-jerasure 2 1 2026-02-22T22:10:17.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:10:17.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T22:10:17.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-22T22:10:17.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T22:10:17.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-22T22:10:17.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T22:10:17.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-22T22:10:17.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T22:10:17.581 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T22:10:17.932 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T22:10:17.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:10:18.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T22:10:18.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:10:18.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:10:18.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:10:18.945 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:10:18.945 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:10:18.945 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:10:18.945 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:10:18.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:10:18.946 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:10:19.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:10:19.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:10:19.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:10:19.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:10:19.053 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:10:19.053 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:10:19.252 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:10:19.253 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:10:19.253 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:10:19.253 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:10:19.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:10:19.253 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:10:19.253 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:10:19.329 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803784 2026-02-22T22:10:19.329 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803784 2026-02-22T22:10:19.329 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784' 2026-02-22T22:10:19.330 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:10:19.330 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:10:19.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607558 2026-02-22T22:10:19.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607558 2026-02-22T22:10:19.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-51539607558' 2026-02-22T22:10:19.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:10:19.404 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:10:19.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=73014444037 2026-02-22T22:10:19.481 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 73014444037 2026-02-22T22:10:19.482 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-51539607558 2-73014444037' 2026-02-22T22:10:19.482 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:10:19.482 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:10:19.559 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182403 2026-02-22T22:10:19.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182403 2026-02-22T22:10:19.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-51539607558 2-73014444037 3-107374182403' 2026-02-22T22:10:19.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:10:19.561 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803784 2026-02-22T22:10:19.561 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:10:19.561 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:10:19.562 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803784 2026-02-22T22:10:19.562 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:10:19.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803784 2026-02-22T22:10:19.564 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803784 2026-02-22T22:10:19.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2026-02-22T22:10:19.565 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:10:19.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803782 -lt 25769803784 2026-02-22T22:10:19.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:10:20.756 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:10:20.757 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:10:20.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803785 -lt 25769803784 2026-02-22T22:10:20.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:10:20.957 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607558 2026-02-22T22:10:20.957 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:10:20.958 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:10:20.959 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607558 2026-02-22T22:10:20.959 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:10:20.960 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607558 2026-02-22T22:10:20.961 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607558' 2026-02-22T22:10:20.961 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607558 2026-02-22T22:10:20.961 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:10:21.156 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607559 -lt 51539607558 2026-02-22T22:10:21.156 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:10:21.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-73014444037 2026-02-22T22:10:21.157 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:10:21.158 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:10:21.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-73014444037 2026-02-22T22:10:21.159 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:10:21.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=73014444037 2026-02-22T22:10:21.161 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 73014444037' 2026-02-22T22:10:21.161 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 73014444037 2026-02-22T22:10:21.161 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:10:21.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 73014444037 -lt 73014444037 2026-02-22T22:10:21.371 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:10:21.372 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182403 2026-02-22T22:10:21.372 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:10:21.374 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:10:21.374 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182403 2026-02-22T22:10:21.375 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:10:21.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182403 2026-02-22T22:10:21.376 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 107374182403 2026-02-22T22:10:21.376 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182403' 2026-02-22T22:10:21.376 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:10:21.579 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182403 -lt 107374182403 2026-02-22T22:10:21.579 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:10:21.580 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:10:21.580 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:10:21.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:10:21.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:10:21.833 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:10:21.833 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:10:21.833 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:10:21.833 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:10:21.834 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:10:21.834 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:10:22.025 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:10:22.026 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:10:22.027 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:10:22.027 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:10:22.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:10:22.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:10:22.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:288: TEST_rados_get_subread_missing: local shard_id=1 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:289: TEST_rados_get_subread_missing: rados_put_get_data remove td/test-erasure-eio 1 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=remove 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-remove-71671-1 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-remove-71671-1 2026-02-22T22:10:22.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:10:22.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:10:22.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-remove-71671-1 2026-02-22T22:10:22.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:10:22.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:10:22.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:10:22.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:10:22.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:10:22.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:10:22.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:10:22.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:10:22.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-remove-71671-1 td/test-erasure-eio/ORIGINAL 2026-02-22T22:10:22.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-71671-1 td/test-erasure-eio 1 2026-02-22T22:10:22.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2026-02-22T22:10:22.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2026-02-22T22:10:22.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2026-02-22T22:10:22.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2026-02-22T22:10:22.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2026-02-22T22:10:22.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2026-02-22T22:10:22.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-71671-1 2026-02-22T22:10:22.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2026-02-22T22:10:22.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2026-02-22T22:10:22.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2026-02-22T22:10:22.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=1 2026-02-22T22:10:22.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2026-02-22T22:10:22.299 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-71671-1 2026-02-22T22:10:22.299 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:10:22.299 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-remove-71671-1 2026-02-22T22:10:22.300 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-remove-71671-1 2026-02-22T22:10:22.300 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:10:22.498 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:10:22.498 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:10:22.498 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:10:22.498 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:10:22.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=1 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 1 obj-remove-71671-1 remove 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-remove-71671-1 remove 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:10:22.500 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:10:23.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:10:23.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-remove-71671-1 remove 2026-02-22T22:10:23.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:10:23.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:10:23.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2026-02-22T22:10:23.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:10:23.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2026-02-22T22:10:23.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-remove-71671-1 remove 2026-02-22T22:10:24.468 INFO:tasks.workunit.client.0.trial034.stdout:remove 1#2:19773d3a:::obj-remove-71671-1:head# 2026-02-22T22:10:25.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2026-02-22T22:10:25.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:10:25.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:10:25.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2026-02-22T22:10:25.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:10:25.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:10:25.006 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:10:25.007 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:25.008 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:25.009 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:10:25.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:10:25.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:10:25.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:10:25.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:10:25.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:10:25.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:10:25.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:10:25.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:10:25.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:10:25.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:10:25.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:10:25.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2026-02-22T22:10:25.011 INFO:tasks.workunit.client.0.trial034.stderr:start osd.1 2026-02-22T22:10:25.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:10:25.011 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2026-02-22T22:10:25.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2026-02-22T22:10:25.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:10:25.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:10:25.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:10:25.031 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:25.030+0000 7fcd6b1038c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:25.032 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:25.030+0000 7fcd6b1038c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:25.034 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:25.030+0000 7fcd6b1038c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:25.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2026-02-22T22:10:25.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:10:25.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:10:25.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:10:25.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:10:25.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:25.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:10:25.206 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:10:25.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:25.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:10:25.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:26.297 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:26.294+0000 7fcd6b1038c0 -1 Falling back to public interface 2026-02-22T22:10:26.408 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:26.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:26.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:10:26.409 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:10:26.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:26.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:10:26.609 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:27.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:27.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:27.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:10:27.610 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:10:27.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:27.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:10:27.808 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:28.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:28.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:28.810 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:10:28.811 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:10:28.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:28.811 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:10:29.010 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:29.305 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:29.302+0000 7fcd6b1038c0 -1 osd.1 32 log_to_monitors true 2026-02-22T22:10:30.011 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:30.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:30.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:10:30.012 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:10:30.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:30.012 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:10:30.212 INFO:tasks.workunit.client.0.trial034.stderr:osd.1 up in weight 1 up_from 36 up_thru 36 down_at 33 last_clean_interval [12,32) [v2:127.0.0.1:6810/1965571068,v1:127.0.0.1:6811/1965571068] [v2:127.0.0.1:6812/1965571068,v1:127.0.0.1:6813/1965571068] exists,up 0eafe266-26e2-4742-885d-3020c8bd1297 2026-02-22T22:10:30.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:10:30.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:10:30.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:10:30.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:10:30.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:10:30.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:10:30.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:10:30.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:10:30.213 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:10:30.213 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:10:30.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:10:30.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:10:30.214 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:10:30.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:10:30.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:10:30.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:10:30.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:10:30.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:10:30.315 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:10:30.519 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:10:30.519 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:10:30.520 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:10:30.520 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:10:30.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:10:30.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:10:30.520 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:10:30.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803789 2026-02-22T22:10:30.597 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803789 2026-02-22T22:10:30.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789' 2026-02-22T22:10:30.598 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:10:30.598 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:10:30.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=154618822659 2026-02-22T22:10:30.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 154618822659 2026-02-22T22:10:30.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-154618822659' 2026-02-22T22:10:30.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:10:30.673 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:10:30.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=73014444041 2026-02-22T22:10:30.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 73014444041 2026-02-22T22:10:30.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-154618822659 2-73014444041' 2026-02-22T22:10:30.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:10:30.750 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:10:30.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182407 2026-02-22T22:10:30.827 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182407 2026-02-22T22:10:30.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803789 1-154618822659 2-73014444041 3-107374182407' 2026-02-22T22:10:30.828 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:10:30.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803789 2026-02-22T22:10:30.828 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:10:30.829 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:10:30.830 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803789 2026-02-22T22:10:30.830 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:10:30.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803789 2026-02-22T22:10:30.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803789' 2026-02-22T22:10:30.832 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803789 2026-02-22T22:10:30.832 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:10:31.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803787 -lt 25769803789 2026-02-22T22:10:31.026 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:10:32.028 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:10:32.028 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:10:32.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803787 -lt 25769803789 2026-02-22T22:10:32.232 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:10:33.234 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:10:33.235 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:10:33.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803789 -lt 25769803789 2026-02-22T22:10:33.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:10:33.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-154618822659 2026-02-22T22:10:33.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:10:33.437 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:10:33.438 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-154618822659 2026-02-22T22:10:33.439 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:10:33.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=154618822659 2026-02-22T22:10:33.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 154618822659' 2026-02-22T22:10:33.440 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 154618822659 2026-02-22T22:10:33.440 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:10:33.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 154618822659 -lt 154618822659 2026-02-22T22:10:33.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:10:33.641 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-73014444041 2026-02-22T22:10:33.641 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:10:33.642 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:10:33.643 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-73014444041 2026-02-22T22:10:33.643 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:10:33.644 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=73014444041 2026-02-22T22:10:33.645 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 73014444041' 2026-02-22T22:10:33.645 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 73014444041 2026-02-22T22:10:33.645 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:10:33.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 73014444042 -lt 73014444041 2026-02-22T22:10:33.848 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:10:33.849 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182407 2026-02-22T22:10:33.849 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:10:33.850 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:10:33.850 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182407 2026-02-22T22:10:33.851 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:10:33.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182407 2026-02-22T22:10:33.852 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182407' 2026-02-22T22:10:33.852 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 107374182407 2026-02-22T22:10:33.853 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:10:34.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182408 -lt 107374182407 2026-02-22T22:10:34.062 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:10:34.062 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:10:34.063 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:10:34.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:10:34.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:10:34.321 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:10:34.321 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:10:34.321 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:10:34.321 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:10:34.321 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:10:34.322 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:10:34.514 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:10:34.514 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:10:34.515 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:10:34.515 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:10:34.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:10:34.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:10:34.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:10:34.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-71671-1 2026-02-22T22:10:34.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:10:34.774 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:10:34.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-71671-1 2026-02-22T22:10:34.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:10:34.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:10:34.775 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-71671-1 td/test-erasure-eio/COPY 2026-02-22T22:10:34.792 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:10:34.793 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:10:34.795 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2026-02-22T22:10:34.795 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2026-02-22T22:10:34.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=2 2026-02-22T22:10:34.797 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-71671-1 td/test-erasure-eio 2 2026-02-22T22:10:34.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2026-02-22T22:10:34.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2026-02-22T22:10:34.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2026-02-22T22:10:34.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2026-02-22T22:10:34.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2026-02-22T22:10:34.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2026-02-22T22:10:34.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-71671-1 2026-02-22T22:10:34.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2026-02-22T22:10:34.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2026-02-22T22:10:34.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2026-02-22T22:10:34.798 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=2 2026-02-22T22:10:34.799 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2026-02-22T22:10:34.799 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-71671-1 2026-02-22T22:10:34.799 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:10:34.799 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-remove-71671-1 2026-02-22T22:10:34.799 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-remove-71671-1 2026-02-22T22:10:34.799 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:10:34.995 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:10:34.996 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:10:34.996 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:10:34.996 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:10:34.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2026-02-22T22:10:34.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2026-02-22T22:10:34.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=2 2026-02-22T22:10:34.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 2 obj-remove-71671-1 remove 2026-02-22T22:10:34.996 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2026-02-22T22:10:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2026-02-22T22:10:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2026-02-22T22:10:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2026-02-22T22:10:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-remove-71671-1 remove 2026-02-22T22:10:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2026-02-22T22:10:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2026-02-22T22:10:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2026-02-22T22:10:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2026-02-22T22:10:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-22T22:10:34.997 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:10:34.997 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:10:34.997 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:10:34.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:10:34.998 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:10:36.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:10:36.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-remove-71671-1 remove 2026-02-22T22:10:36.305 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2026-02-22T22:10:36.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2026-02-22T22:10:36.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2026-02-22T22:10:36.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2026-02-22T22:10:36.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2026-02-22T22:10:36.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-remove-71671-1 remove 2026-02-22T22:10:36.960 INFO:tasks.workunit.client.0.trial034.stdout:remove 2#2:19773d3a:::obj-remove-71671-1:head# 2026-02-22T22:10:37.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2026-02-22T22:10:37.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:10:37.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:10:37.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T22:10:37.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:10:37.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:10:37.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:10:37.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:10:37.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:10:37.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:10:37.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:10:37.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:10:37.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:10:37.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:10:37.499 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:10:37.499 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:10:37.499 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:10:37.499 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:10:37.501 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:10:37.502 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:37.502 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:37.502 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:10:37.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:10:37.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:10:37.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:10:37.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:10:37.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:10:37.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:10:37.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:10:37.502 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:10:37.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:10:37.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:10:37.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:10:37.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T22:10:37.503 INFO:tasks.workunit.client.0.trial034.stderr:start osd.2 2026-02-22T22:10:37.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:10:37.503 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-22T22:10:37.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T22:10:37.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:10:37.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:10:37.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:10:37.522 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:37.518+0000 7f4de97298c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:37.524 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:37.522+0000 7f4de97298c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:37.525 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:37.522+0000 7f4de97298c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:37.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T22:10:37.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:10:37.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:10:37.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:10:37.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:10:37.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:37.706 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:10:37.706 INFO:tasks.workunit.client.0.trial034.stderr:0 2026-02-22T22:10:37.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:37.707 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:10:37.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:38.761 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:38.758+0000 7f4de97298c0 -1 Falling back to public interface 2026-02-22T22:10:38.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:38.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:38.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:10:38.902 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:10:38.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:38.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:10:39.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:40.097 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:40.097 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:40.097 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:10:40.098 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:10:40.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:40.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:10:40.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:41.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:41.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:41.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:10:41.294 INFO:tasks.workunit.client.0.trial034.stderr:3 2026-02-22T22:10:41.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:41.294 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:10:41.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:41.775 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:41.774+0000 7f4de97298c0 -1 osd.2 39 log_to_monitors true 2026-02-22T22:10:42.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:42.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:42.487 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:10:42.487 INFO:tasks.workunit.client.0.trial034.stderr:4 2026-02-22T22:10:42.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:42.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:10:42.682 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 up in weight 1 up_from 43 up_thru 33 down_at 40 last_clean_interval [17,39) [v2:127.0.0.1:6818/17359695,v1:127.0.0.1:6819/17359695] [v2:127.0.0.1:6820/17359695,v1:127.0.0.1:6821/17359695] exists,up a28e3b93-ac60-4cab-a420-8bd96b96c2dd 2026-02-22T22:10:42.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:10:42.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:10:42.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:10:42.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2026-02-22T22:10:42.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:10:42.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:10:42.683 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:10:42.683 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:10:42.684 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:10:42.684 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:10:42.684 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:10:42.684 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:10:42.684 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:10:42.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:10:42.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:10:42.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:10:42.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:10:42.791 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:10:42.791 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:10:42.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:10:42.988 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:10:42.988 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:10:42.988 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:10:42.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:10:42.988 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:10:42.988 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:10:43.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803793 2026-02-22T22:10:43.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803793 2026-02-22T22:10:43.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793' 2026-02-22T22:10:43.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:10:43.062 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:10:43.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=154618822663 2026-02-22T22:10:43.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 154618822663 2026-02-22T22:10:43.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-154618822663' 2026-02-22T22:10:43.136 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:10:43.136 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:10:43.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=184683593731 2026-02-22T22:10:43.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 184683593731 2026-02-22T22:10:43.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-154618822663 2-184683593731' 2026-02-22T22:10:43.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:10:43.215 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:10:43.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=107374182412 2026-02-22T22:10:43.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 107374182412 2026-02-22T22:10:43.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803793 1-154618822663 2-184683593731 3-107374182412' 2026-02-22T22:10:43.291 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:10:43.291 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803793 2026-02-22T22:10:43.291 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:10:43.293 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:10:43.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803793 2026-02-22T22:10:43.294 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:10:43.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803793 2026-02-22T22:10:43.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803793' 2026-02-22T22:10:43.296 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.0 seq 25769803793 2026-02-22T22:10:43.296 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:10:43.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803791 -lt 25769803793 2026-02-22T22:10:43.488 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:10:44.489 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:10:44.490 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:10:44.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803793 -lt 25769803793 2026-02-22T22:10:44.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:10:44.690 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-154618822663 2026-02-22T22:10:44.690 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:10:44.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:10:44.691 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-154618822663 2026-02-22T22:10:44.692 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:10:44.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=154618822663 2026-02-22T22:10:44.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 154618822663' 2026-02-22T22:10:44.693 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.1 seq 154618822663 2026-02-22T22:10:44.694 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:10:44.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 154618822664 -lt 154618822663 2026-02-22T22:10:44.897 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:10:44.898 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-184683593731 2026-02-22T22:10:44.898 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:10:44.899 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:10:44.900 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-184683593731 2026-02-22T22:10:44.900 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:10:44.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=184683593731 2026-02-22T22:10:44.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 184683593731' 2026-02-22T22:10:44.901 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.2 seq 184683593731 2026-02-22T22:10:44.902 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:10:45.111 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 184683593731 -lt 184683593731 2026-02-22T22:10:45.111 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:10:45.112 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-107374182412 2026-02-22T22:10:45.112 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:10:45.113 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:10:45.114 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-107374182412 2026-02-22T22:10:45.114 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:10:45.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=107374182412 2026-02-22T22:10:45.115 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 107374182412' 2026-02-22T22:10:45.115 INFO:tasks.workunit.client.0.trial034.stderr:waiting osd.3 seq 107374182412 2026-02-22T22:10:45.116 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:10:45.315 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 107374182412 -lt 107374182412 2026-02-22T22:10:45.316 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:10:45.316 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:10:45.316 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:10:45.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:10:45.587 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:10:45.587 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:10:45.588 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:10:45.588 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:10:45.588 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:10:45.588 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:10:45.588 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:10:45.786 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:10:45.787 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:10:45.787 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:10:45.787 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:10:46.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:10:46.049 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:10:46.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:10:46.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-71671-1 fail 2026-02-22T22:10:46.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:10:46.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:10:46.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-71671-1 2026-02-22T22:10:46.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2026-02-22T22:10:46.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2026-02-22T22:10:46.050 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-remove-71671-1 td/test-erasure-eio/COPY 2026-02-22T22:10:46.066 INFO:tasks.workunit.client.0.trial034.stderr:error getting pool-jerasure/obj-remove-71671-1: (5) Input/output error 2026-02-22T22:10:46.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2026-02-22T22:10:46.068 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2026-02-22T22:10:46.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:290: TEST_rados_get_subread_missing: delete_erasure_coded_pool pool-jerasure 2026-02-22T22:10:46.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:10:46.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T22:10:46.307 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T22:10:46.316 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T22:10:46.575 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T22:10:46.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T22:10:46.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:10:46.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:10:46.585 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:10:46.585 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:10:46.585 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:10:46.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:10:46.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:10:46.586 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:10:46.696 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:10:46.696 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:10:46.697 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:10:46.697 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:10:46.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:10:46.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:10:46.699 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:10:46.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:10:46.700 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:10:46.701 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:10:46.701 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:10:46.702 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:10:46.703 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:10:46.703 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:10:46.718 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:10:46.718 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:46.718 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:46.718 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:10:46.720 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:10:46.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:10:46.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T22:10:46.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T22:10:46.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T22:10:46.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T22:10:46.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:10:46.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:10:46.721 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:10:46.721 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:10:46.722 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:10:46.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:10:46.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:10:46.722 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:10:46.723 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:10:46.723 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:10:46.724 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:10:46.724 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:10:46.725 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:10:46.725 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:10:46.726 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:10:46.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:10:46.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:10:46.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:10:46.728 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:10:46.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:10:46.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:10:46.731 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:10:46.732 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:10:46.732 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:46.732 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:46.732 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:10:46.733 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:10:46.734 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:10:46.734 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T22:10:46.735 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:10:46.735 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:46.736 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:46.736 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T22:10:46.739 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:10:46.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:10:46.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:10:46.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T22:10:46.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T22:10:46.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T22:10:46.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:10:46.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:10:46.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:10:46.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T22:10:46.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T22:10:46.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:10:46.760 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:10:46.760 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:10:46.760 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:10:46.760 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:46.760 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:46.760 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:10:46.761 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:10:46.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:10:46.791 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:10:46.791 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:10:46.791 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:10:46.792 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:10:46.792 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:10:46.793 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:10:46.793 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:10:46.793 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:10:46.794 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:10:46.794 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:46.794 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:46.794 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:10:46.794 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:10:46.794 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T22:10:46.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:10:46.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:10:46.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:10:46.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:10:46.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:10:46.848 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:10:46.848 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:10:46.848 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:10:46.849 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:10:46.849 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:46.849 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:46.849 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:10:46.849 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:10:46.849 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T22:10:46.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T22:10:46.904 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T22:10:46.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:10:46.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:10:46.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:10:46.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T22:10:46.905 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:10:47.013 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:10:47.014 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:10:47.014 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:10:47.014 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:10:47.014 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:47.014 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:47.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:10:47.015 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:10:47.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:10:47.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T22:10:47.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:10:47.140 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T22:10:47.149 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:10:48.151 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T22:10:48.151 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:10:48.151 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:10:48.151 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:10:48.151 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:48.151 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:48.151 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:10:48.151 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T22:10:48.151 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T22:10:48.201 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:10:48.206 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T22:10:48.207 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:10:46.778+0000 7fdd05d8fec0 0 load: jerasure load: lrc 2026-02-22T22:10:48.208 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_0 td/test-erasure-eio 2026-02-22T22:10:48.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:330: TEST_rados_get_with_subreadall_eio_shard_0: local dir=td/test-erasure-eio 2026-02-22T22:10:48.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:331: TEST_rados_get_with_subreadall_eio_shard_0: local shard_id=0 2026-02-22T22:10:48.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:333: TEST_rados_get_with_subreadall_eio_shard_0: setup_osds 4 2026-02-22T22:10:48.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-22T22:10:48.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T22:10:48.209 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-22T22:10:48.210 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-22T22:10:48.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:10:48.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T22:10:48.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:10:48.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:10:48.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:10:48.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:10:48.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T22:10:48.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:10:48.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:10:48.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:10:48.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:10:48.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T22:10:48.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T22:10:48.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:10:48.214 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:10:48.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:10:48.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:10:48.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:10:48.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:10:48.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:10:48.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:10:48.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:10:48.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:10:48.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T22:10:48.217 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:10:48.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=57e7c401-a265-4691-aa0d-9b09744bca41 2026-02-22T22:10:48.217 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 57e7c401-a265-4691-aa0d-9b09744bca41' 2026-02-22T22:10:48.217 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 57e7c401-a265-4691-aa0d-9b09744bca41 2026-02-22T22:10:48.218 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:10:48.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDofptpSUaQDRAABzq6jti5XuB6tCJ9jZIX/A== 2026-02-22T22:10:48.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDofptpSUaQDRAABzq6jti5XuB6tCJ9jZIX/A=="}' 2026-02-22T22:10:48.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 57e7c401-a265-4691-aa0d-9b09744bca41 -i td/test-erasure-eio/0/new.json 2026-02-22T22:10:48.337 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:10:48.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T22:10:48.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDofptpSUaQDRAABzq6jti5XuB6tCJ9jZIX/A== --osd-uuid 57e7c401-a265-4691-aa0d-9b09744bca41 2026-02-22T22:10:48.360 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:48.358+0000 7fee7cffe8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:48.361 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:48.358+0000 7fee7cffe8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:48.362 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:48.362+0000 7fee7cffe8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:48.363 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:48.362+0000 7fee7cffe8c0 -1 bdev(0x5644b474dc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:10:48.363 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:48.362+0000 7fee7cffe8c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T22:10:50.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T22:10:50.665 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:10:50.666 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T22:10:50.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:10:50.667 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:10:50.942 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T22:10:50.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:10:50.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:10:50.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:10:50.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:10:50.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:10:50.966 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:50.962+0000 7f43dcb778c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:50.968 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:50.966+0000 7f43dcb778c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:50.969 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:50.966+0000 7f43dcb778c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:51.144 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:10:51.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:10:51.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:10:51.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:10:51.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:10:51.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:51.145 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:10:51.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:10:51.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:51.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:10:51.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:52.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:52.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:52.346 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:10:52.346 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:10:52.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:52.347 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:10:52.488 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:52.486+0000 7f43dcb778c0 -1 Falling back to public interface 2026-02-22T22:10:52.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:53.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:53.547 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:53.547 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:10:53.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:10:53.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:53.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:10:53.746 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:54.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:54.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:54.748 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:10:54.749 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:10:54.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:54.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:10:54.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:55.535 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:55.534+0000 7f43dcb778c0 -1 osd.0 0 log_to_monitors true 2026-02-22T22:10:55.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:55.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:55.946 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:10:55.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:10:55.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:55.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:10:56.145 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:10:57.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:10:57.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:10:57.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:10:57.147 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:10:57.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:10:57.148 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:10:57.340 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/464673577,v1:127.0.0.1:6803/464673577] [v2:127.0.0.1:6804/464673577,v1:127.0.0.1:6805/464673577] exists,up 57e7c401-a265-4691-aa0d-9b09744bca41 2026-02-22T22:10:57.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:10:57.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:10:57.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:10:57.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:10:57.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T22:10:57.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:10:57.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:10:57.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:10:57.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:10:57.341 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:10:57.342 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:10:57.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:10:57.344 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:10:57.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c4c43cf5-8db3-4f19-a69f-4cbaa7a38cbe 2026-02-22T22:10:57.345 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c4c43cf5-8db3-4f19-a69f-4cbaa7a38cbe' 2026-02-22T22:10:57.345 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 c4c43cf5-8db3-4f19-a69f-4cbaa7a38cbe 2026-02-22T22:10:57.345 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:10:57.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDxfptpFexMFRAADR4cb3QKqz9glez5X3lCOw== 2026-02-22T22:10:57.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDxfptpFexMFRAADR4cb3QKqz9glez5X3lCOw=="}' 2026-02-22T22:10:57.356 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c4c43cf5-8db3-4f19-a69f-4cbaa7a38cbe -i td/test-erasure-eio/1/new.json 2026-02-22T22:10:57.566 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:10:57.576 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T22:10:57.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxfptpFexMFRAADR4cb3QKqz9glez5X3lCOw== --osd-uuid c4c43cf5-8db3-4f19-a69f-4cbaa7a38cbe 2026-02-22T22:10:57.593 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:57.590+0000 7f92f1dcc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:57.594 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:57.590+0000 7f92f1dcc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:57.596 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:57.594+0000 7f92f1dcc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:10:57.596 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:57.594+0000 7f92f1dcc8c0 -1 bdev(0x558530dfbc00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:10:57.596 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:10:57.594+0000 7f92f1dcc8c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T22:10:59.860 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T22:10:59.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:10:59.861 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T22:10:59.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:10:59.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:11:00.132 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T22:11:00.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:11:00.133 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:11:00.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:11:00.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:11:00.134 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:11:00.155 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:00.154+0000 7f795a48c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:00.156 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:00.154+0000 7f795a48c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:00.158 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:00.154+0000 7f795a48c8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:00.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:11:00.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:11:00.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:11:00.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:11:00.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:11:00.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:00.327 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:11:00.327 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:11:00.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:00.328 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:11:00.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:01.526 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:01.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:01.527 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:11:01.527 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:11:01.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:01.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:11:01.727 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:01.881 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:01.877+0000 7f795a48c8c0 -1 Falling back to public interface 2026-02-22T22:11:02.728 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:02.729 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:02.729 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:11:02.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:11:02.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:02.730 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:11:02.931 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:03.932 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:03.933 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:03.933 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:11:03.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:11:03.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:03.934 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:11:04.138 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:04.897 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:04.893+0000 7f795a48c8c0 -1 osd.1 0 log_to_monitors true 2026-02-22T22:11:05.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:05.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:05.139 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:11:05.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:11:05.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:05.139 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:11:05.348 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:06.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:06.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:06.350 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:11:06.351 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:11:06.351 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:06.351 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:11:06.549 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/244169527,v1:127.0.0.1:6811/244169527] [v2:127.0.0.1:6812/244169527,v1:127.0.0.1:6813/244169527] exists,up c4c43cf5-8db3-4f19-a69f-4cbaa7a38cbe 2026-02-22T22:11:06.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:11:06.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:11:06.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:11:06.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:11:06.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T22:11:06.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:11:06.550 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:11:06.551 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:11:06.552 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:11:06.553 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:11:06.553 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=38bdb0e3-21af-4ee0-a030-c23e714d5640 2026-02-22T22:11:06.554 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 38bdb0e3-21af-4ee0-a030-c23e714d5640' 2026-02-22T22:11:06.554 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 38bdb0e3-21af-4ee0-a030-c23e714d5640 2026-02-22T22:11:06.554 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:11:06.565 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6fptpCKjEIRAAKlZS/W2oFZQ/0p0Hs4GUAA== 2026-02-22T22:11:06.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6fptpCKjEIRAAKlZS/W2oFZQ/0p0Hs4GUAA=="}' 2026-02-22T22:11:06.566 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 38bdb0e3-21af-4ee0-a030-c23e714d5640 -i td/test-erasure-eio/2/new.json 2026-02-22T22:11:06.768 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:11:06.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T22:11:06.781 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6fptpCKjEIRAAKlZS/W2oFZQ/0p0Hs4GUAA== --osd-uuid 38bdb0e3-21af-4ee0-a030-c23e714d5640 2026-02-22T22:11:06.805 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:06.801+0000 7f484a2458c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:06.807 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:06.805+0000 7f484a2458c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:06.808 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:06.805+0000 7f484a2458c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:06.808 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:06.805+0000 7f484a2458c0 -1 bdev(0x557d287e9c00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:11:06.809 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:06.805+0000 7f484a2458c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T22:11:09.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T22:11:09.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:11:09.081 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T22:11:09.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:11:09.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:11:09.355 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T22:11:09.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:11:09.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:11:09.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:11:09.357 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:11:09.358 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:11:09.372 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:09.369+0000 7f2abe16f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:09.373 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:09.369+0000 7f2abe16f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:09.374 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:09.369+0000 7f2abe16f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:09.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:11:09.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:11:09.548 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:11:09.549 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:11:09.549 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:11:09.549 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:09.549 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:11:09.549 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:11:09.549 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:09.549 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:11:09.747 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:10.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:10.749 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:10.750 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:11:10.750 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:11:10.751 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:10.751 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:11:10.888 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:10.885+0000 7f2abe16f8c0 -1 Falling back to public interface 2026-02-22T22:11:10.949 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:11.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:11.950 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:11.951 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:11:11.951 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:11:11.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:11.952 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:11:12.146 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:13.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:13.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:13.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:11:13.147 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:11:13.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:13.147 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:11:13.343 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:13.917 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:13.913+0000 7f2abe16f8c0 -1 osd.2 0 log_to_monitors true 2026-02-22T22:11:14.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:14.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:14.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:11:14.344 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:11:14.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:14.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:11:14.541 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2953856959,v1:127.0.0.1:6819/2953856959] [v2:127.0.0.1:6820/2953856959,v1:127.0.0.1:6821/2953856959] exists,up 38bdb0e3-21af-4ee0-a030-c23e714d5640 2026-02-22T22:11:14.542 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:11:14.542 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:11:14.543 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:11:14.544 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:11:14.544 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:14.544 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:14.544 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:14.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:14.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:11:14.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:11:14.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:11:14.544 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:11:14.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:11:14.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:11:14.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:11:14.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:11:14.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:11:14.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:11:14.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:11:14.545 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:11:14.545 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:11:14.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=749ba3bb-eb74-41c5-a94b-2a200c3b2981 2026-02-22T22:11:14.546 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 749ba3bb-eb74-41c5-a94b-2a200c3b2981 2026-02-22T22:11:14.546 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 749ba3bb-eb74-41c5-a94b-2a200c3b2981' 2026-02-22T22:11:14.546 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:11:14.558 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACf5tp+I1OIRAAnRUnNBvqqNGC4qBXLX1Ebg== 2026-02-22T22:11:14.558 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACf5tp+I1OIRAAnRUnNBvqqNGC4qBXLX1Ebg=="}' 2026-02-22T22:11:14.558 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 749ba3bb-eb74-41c5-a94b-2a200c3b2981 -i td/test-erasure-eio/3/new.json 2026-02-22T22:11:14.768 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:11:14.778 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T22:11:14.779 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACf5tp+I1OIRAAnRUnNBvqqNGC4qBXLX1Ebg== --osd-uuid 749ba3bb-eb74-41c5-a94b-2a200c3b2981 2026-02-22T22:11:14.796 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:14.793+0000 7f2a59e488c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:14.797 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:14.793+0000 7f2a59e488c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:14.798 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:14.793+0000 7f2a59e488c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:14.799 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:14.797+0000 7f2a59e488c0 -1 bdev(0x5634cc0a1c00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T22:11:14.799 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:14.797+0000 7f2a59e488c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T22:11:17.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T22:11:17.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:11:17.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T22:11:17.081 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T22:11:17.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:11:17.348 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T22:11:17.349 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T22:11:17.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:11:17.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:11:17.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:11:17.350 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:11:17.364 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:17.361+0000 7f18c03798c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:17.366 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:17.361+0000 7f18c03798c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:17.367 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:17.365+0000 7f18c03798c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:17.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T22:11:17.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:11:17.542 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:11:17.542 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:11:17.542 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:11:17.542 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:17.542 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:11:17.542 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:11:17.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:17.543 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:11:17.739 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:11:18.740 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:11:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:18.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:11:18.892 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:18.889+0000 7f18c03798c0 -1 Falling back to public interface 2026-02-22T22:11:18.942 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:19.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:19.943 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:19.944 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:11:19.944 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:11:19.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:19.945 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:11:20.140 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:21.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:21.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:21.141 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:11:21.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:11:21.141 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:21.142 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:11:21.340 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:21.910 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:21.905+0000 7f18c03798c0 -1 osd.3 0 log_to_monitors true 2026-02-22T22:11:22.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:22.342 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:22.343 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:11:22.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:11:22.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:22.344 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:11:22.538 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/67731399,v1:127.0.0.1:6827/67731399] [v2:127.0.0.1:6828/67731399,v1:127.0.0.1:6829/67731399] exists,up 749ba3bb-eb74-41c5-a94b-2a200c3b2981 2026-02-22T22:11:22.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:11:22.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:11:22.540 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:11:22.540 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T22:11:22.540 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:11:22.540 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:11:22.540 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:11:22.540 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:22.540 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:22.541 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T22:11:22.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T22:11:22.541 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T22:11:22.601 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:11:22.607 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T22:11:22.609 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:10:52.490+0000 7f43dcb778c0 0 load: jerasure load: lrc 2026-02-22T22:11:22.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:335: TEST_rados_get_with_subreadall_eio_shard_0: local poolname=pool-jerasure 2026-02-22T22:11:22.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:336: TEST_rados_get_with_subreadall_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2026-02-22T22:11:22.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:11:22.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T22:11:22.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-22T22:11:22.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T22:11:22.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-22T22:11:22.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T22:11:22.611 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-22T22:11:22.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T22:11:22.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T22:11:23.286 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T22:11:23.295 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:11:24.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T22:11:24.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:11:24.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:11:24.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:11:24.298 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:11:24.298 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:11:24.298 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:11:24.299 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:11:24.299 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:11:24.299 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:11:24.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:11:24.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:11:24.404 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:11:24.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:11:24.405 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:11:24.405 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:11:24.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:11:24.612 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:11:24.612 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:11:24.612 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:11:24.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:11:24.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:11:24.613 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:11:24.686 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803784 2026-02-22T22:11:24.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803784 2026-02-22T22:11:24.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784' 2026-02-22T22:11:24.687 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:11:24.687 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:11:24.762 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607558 2026-02-22T22:11:24.762 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607558 2026-02-22T22:11:24.762 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-51539607558' 2026-02-22T22:11:24.762 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:11:24.762 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:11:24.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=73014444036 2026-02-22T22:11:24.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 73014444036 2026-02-22T22:11:24.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-51539607558 2-73014444036' 2026-02-22T22:11:24.837 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:11:24.838 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:11:24.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=103079215107 2026-02-22T22:11:24.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 103079215107 2026-02-22T22:11:24.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-51539607558 2-73014444036 3-103079215107' 2026-02-22T22:11:24.912 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:11:24.912 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803784 2026-02-22T22:11:24.913 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:11:24.914 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:11:24.915 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803784 2026-02-22T22:11:24.915 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:11:24.916 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803784 2026-02-22T22:11:24.917 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2026-02-22T22:11:24.917 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803784 2026-02-22T22:11:24.917 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:11:25.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803782 -lt 25769803784 2026-02-22T22:11:25.127 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:11:26.127 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:11:26.128 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:11:26.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803784 -lt 25769803784 2026-02-22T22:11:26.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:11:26.321 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607558 2026-02-22T22:11:26.322 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:11:26.323 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:11:26.324 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607558 2026-02-22T22:11:26.324 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:11:26.325 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607558 2026-02-22T22:11:26.326 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607558 2026-02-22T22:11:26.326 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607558' 2026-02-22T22:11:26.326 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:11:26.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607558 -lt 51539607558 2026-02-22T22:11:26.528 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:11:26.529 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-73014444036 2026-02-22T22:11:26.529 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:11:26.530 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:11:26.531 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-73014444036 2026-02-22T22:11:26.531 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:11:26.533 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=73014444036 2026-02-22T22:11:26.533 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 73014444036 2026-02-22T22:11:26.534 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 73014444036' 2026-02-22T22:11:26.534 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:11:26.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 73014444037 -lt 73014444036 2026-02-22T22:11:26.740 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:11:26.741 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-103079215107 2026-02-22T22:11:26.741 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:11:26.742 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:11:26.743 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-103079215107 2026-02-22T22:11:26.743 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:11:26.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=103079215107 2026-02-22T22:11:26.745 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 103079215107' 2026-02-22T22:11:26.745 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 103079215107 2026-02-22T22:11:26.745 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:11:26.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 103079215107 -lt 103079215107 2026-02-22T22:11:26.956 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:11:26.957 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:11:26.957 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:11:27.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:11:27.219 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:11:27.220 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:11:27.220 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:11:27.220 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:11:27.220 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:11:27.220 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:11:27.220 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:11:27.414 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:11:27.415 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:11:27.415 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:11:27.415 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:11:27.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:11:27.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:11:27.671 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:11:27.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:338: TEST_rados_get_with_subreadall_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 recovery 2026-02-22T22:11:27.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2026-02-22T22:11:27.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-22T22:11:27.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-22T22:11:27.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-22T22:11:27.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2026-02-22T22:11:27.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-22T22:11:27.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2026-02-22T22:11:27.672 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-22T22:11:27.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-71671-0 2026-02-22T22:11:27.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-71671-0 2026-02-22T22:11:27.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:11:27.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:11:27.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-71671-0 2026-02-22T22:11:27.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:11:27.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:11:27.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:11:27.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:11:27.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:11:27.673 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:11:27.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:11:27.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:11:27.674 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-71671-0 td/test-erasure-eio/ORIGINAL 2026-02-22T22:11:27.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-71671-0 td/test-erasure-eio 0 2026-02-22T22:11:27.691 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T22:11:27.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T22:11:27.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T22:11:27.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T22:11:27.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T22:11:27.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T22:11:27.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=obj-eio-71671-0 2026-02-22T22:11:27.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T22:11:27.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T22:11:27.692 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T22:11:27.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=0 2026-02-22T22:11:27.693 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T22:11:27.693 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure obj-eio-71671-0 2026-02-22T22:11:27.693 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:11:27.693 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-71671-0 2026-02-22T22:11:27.693 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-71671-0 2026-02-22T22:11:27.694 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:11:27.890 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:11:27.890 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:11:27.890 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:11:27.890 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:11:27.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '1' '2') 2026-02-22T22:11:27.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T22:11:27.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=3 2026-02-22T22:11:27.891 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T22:11:27.891 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/3/type 2026-02-22T22:11:27.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T22:11:27.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2026-02-22T22:11:27.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T22:11:27.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2026-02-22T22:11:27.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T22:11:27.892 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T22:11:27.893 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T22:11:27.893 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2026-02-22T22:11:27.893 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T22:11:27.893 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T22:11:27.893 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:11:27.894 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:27.894 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:27.894 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T22:11:27.894 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2026-02-22T22:11:27.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T22:11:27.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T22:11:27.954 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T22:11:27.955 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.3 2026-02-22T22:11:27.955 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2026-02-22T22:11:27.955 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2026-02-22T22:11:27.955 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:11:27.955 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:27.955 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:27.955 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.3.asok 2026-02-22T22:11:27.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T22:11:27.956 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-71671-0 0 2026-02-22T22:11:28.013 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-71671-0 2026-02-22T22:11:28.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:11:28.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:11:28.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-71671-0 2026-02-22T22:11:28.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:11:28.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:11:28.014 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-71671-0 td/test-erasure-eio/COPY 2026-02-22T22:11:28.033 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:11:28.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:11:28.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2026-02-22T22:11:28.036 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-71671-0 2026-02-22T22:11:28.036 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:11:28.036 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-71671-0 2026-02-22T22:11:28.036 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-71671-0 2026-02-22T22:11:28.037 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:11:28.244 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:11:28.244 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:11:28.244 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:11:28.244 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:11:28.244 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2026-02-22T22:11:28.244 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2026-02-22T22:11:28.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2026-02-22T22:11:28.245 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-22T22:11:28.245 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:11:28.245 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:11:28.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:11:28.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:11:28.246 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:11:29.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:11:29.555 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2026-02-22T22:11:29.804 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 is already out. 2026-02-22T22:11:29.815 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-71671-0 2026-02-22T22:11:29.815 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:11:29.815 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-71671-0 2026-02-22T22:11:29.815 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2026-02-22T22:11:29.816 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-71671-0 2026-02-22T22:11:29.816 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:11:30.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:11:30.016 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:11:30.016 INFO:tasks.workunit.client.0.trial034.stderr:0' 2026-02-22T22:11:30.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2026-02-22T22:11:30.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2026-02-22T22:11:30.264 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 is already in. 2026-02-22T22:11:30.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2026-02-22T22:11:30.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:11:30.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:11:30.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T22:11:30.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:11:30.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:11:30.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:11:30.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:11:30.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:11:30.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:11:30.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:11:30.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:11:30.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:11:30.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:11:30.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:11:30.278 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:11:30.278 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:11:30.278 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:11:30.278 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:11:30.279 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:30.279 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:30.279 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:30.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:30.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:11:30.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:11:30.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:11:30.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:11:30.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:11:30.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:11:30.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:11:30.279 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:11:30.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:11:30.280 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:11:30.280 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T22:11:30.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T22:11:30.281 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:11:30.281 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-22T22:11:30.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T22:11:30.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:11:30.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:11:30.282 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:11:30.300 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:30.297+0000 7f2b3317f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:30.302 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:30.297+0000 7f2b3317f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:30.302 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:30.297+0000 7f2b3317f8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:30.476 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T22:11:30.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:11:30.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:11:30.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:11:30.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:11:30.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:30.477 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:11:30.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:11:30.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:30.477 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:11:30.677 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:31.573 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:31.569+0000 7f2b3317f8c0 -1 Falling back to public interface 2026-02-22T22:11:31.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:31.678 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:31.679 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:11:31.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:11:31.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:31.679 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:11:31.878 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:32.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:32.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:32.879 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:11:32.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:11:32.879 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:32.880 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:11:33.077 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:34.079 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:34.079 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:34.079 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:11:34.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:11:34.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:34.080 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:11:34.276 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:34.819 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:34.817+0000 7f2b3317f8c0 -1 osd.2 32 log_to_monitors true 2026-02-22T22:11:35.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:35.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:35.277 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:11:35.278 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:11:35.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:35.278 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:11:35.475 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 41 up_thru 17 down_at 33 last_clean_interval [17,32) [v2:127.0.0.1:6818/1669906397,v1:127.0.0.1:6819/1669906397] [v2:127.0.0.1:6820/1669906397,v1:127.0.0.1:6821/1669906397] exists,up 38bdb0e3-21af-4ee0-a030-c23e714d5640 2026-02-22T22:11:35.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:11:35.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:11:35.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:11:35.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2026-02-22T22:11:35.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:11:35.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:11:35.476 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:11:35.476 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:11:35.476 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:11:35.476 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:11:35.477 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:11:35.477 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:11:35.477 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:11:35.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:11:35.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:11:35.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:11:35.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:11:35.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:11:35.589 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:11:35.789 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:11:35.789 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:11:35.790 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:11:35.790 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:11:35.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:11:35.790 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:11:35.790 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:11:35.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803788 2026-02-22T22:11:35.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803788 2026-02-22T22:11:35.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803788' 2026-02-22T22:11:35.865 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:11:35.865 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:11:35.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=51539607562 2026-02-22T22:11:35.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 51539607562 2026-02-22T22:11:35.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803788 1-51539607562' 2026-02-22T22:11:35.940 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:11:35.941 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:11:36.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=176093659139 2026-02-22T22:11:36.016 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 176093659139 2026-02-22T22:11:36.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803788 1-51539607562 2-176093659139' 2026-02-22T22:11:36.017 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:11:36.017 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:11:36.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=103079215111 2026-02-22T22:11:36.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 103079215111 2026-02-22T22:11:36.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803788 1-51539607562 2-176093659139 3-103079215111' 2026-02-22T22:11:36.093 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:11:36.094 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803788 2026-02-22T22:11:36.094 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:11:36.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:11:36.096 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803788 2026-02-22T22:11:36.096 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:11:36.097 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803788 2026-02-22T22:11:36.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803788' 2026-02-22T22:11:36.098 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803788 2026-02-22T22:11:36.098 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:11:36.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803786 -lt 25769803788 2026-02-22T22:11:36.302 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:11:37.303 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:11:37.304 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:11:37.503 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803789 -lt 25769803788 2026-02-22T22:11:37.504 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:11:37.504 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-51539607562 2026-02-22T22:11:37.504 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:11:37.506 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:11:37.506 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-51539607562 2026-02-22T22:11:37.507 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:11:37.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=51539607562 2026-02-22T22:11:37.508 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 51539607562' 2026-02-22T22:11:37.509 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 51539607562 2026-02-22T22:11:37.509 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:11:37.710 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 51539607563 -lt 51539607562 2026-02-22T22:11:37.711 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:11:37.711 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-176093659139 2026-02-22T22:11:37.711 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:11:37.712 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:11:37.713 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-176093659139 2026-02-22T22:11:37.713 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:11:37.714 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=176093659139 2026-02-22T22:11:37.715 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 176093659139 2026-02-22T22:11:37.715 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 176093659139' 2026-02-22T22:11:37.715 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:11:37.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 176093659139 -lt 176093659139 2026-02-22T22:11:37.922 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:11:37.922 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-103079215111 2026-02-22T22:11:37.923 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:11:37.924 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:11:37.925 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-103079215111 2026-02-22T22:11:37.925 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:11:37.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=103079215111 2026-02-22T22:11:37.926 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 103079215111 2026-02-22T22:11:37.926 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 103079215111' 2026-02-22T22:11:37.927 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:11:38.126 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 103079215111 -lt 103079215111 2026-02-22T22:11:38.127 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:11:38.127 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:11:38.127 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:11:38.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:11:38.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:11:38.381 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:11:38.381 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:11:38.381 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:11:38.381 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:11:38.381 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:11:38.381 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:11:38.570 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:11:38.570 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:11:38.571 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:11:38.571 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:11:38.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:11:38.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:11:38.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:11:38.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:340: TEST_rados_get_with_subreadall_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2026-02-22T22:11:38.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:11:38.824 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T22:11:39.038 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T22:11:39.048 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T22:11:39.311 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T22:11:39.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T22:11:39.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:11:39.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:11:39.320 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:11:39.321 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:11:39.321 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:11:39.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:11:39.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:11:39.321 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:11:39.432 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:11:39.433 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:11:39.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:11:39.434 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:11:39.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:11:39.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:11:39.436 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:11:39.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:11:39.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:11:39.438 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:11:39.438 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:11:39.439 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:11:39.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:11:39.440 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:11:39.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:11:39.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:39.455 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:39.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:11:39.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:11:39.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:11:39.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2026-02-22T22:11:39.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2026-02-22T22:11:39.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2026-02-22T22:11:39.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2026-02-22T22:11:39.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:11:39.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:11:39.459 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:11:39.459 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:11:39.460 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:11:39.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:11:39.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:11:39.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:11:39.460 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:11:39.460 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:11:39.461 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:11:39.462 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:11:39.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:11:39.463 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:11:39.463 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:11:39.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:11:39.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:11:39.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:11:39.465 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:11:39.466 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:11:39.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:11:39.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:11:39.469 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:11:39.469 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:39.469 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:39.470 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:11:39.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:11:39.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:11:39.471 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2026-02-22T22:11:39.472 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2026-02-22T22:11:39.472 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:39.472 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:39.472 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71671 2026-02-22T22:11:39.474 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2026-02-22T22:11:39.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2026-02-22T22:11:39.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2026-02-22T22:11:39.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2026-02-22T22:11:39.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2026-02-22T22:11:39.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2026-02-22T22:11:39.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2026-02-22T22:11:39.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2026-02-22T22:11:39.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2026-02-22T22:11:39.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2026-02-22T22:11:39.475 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2026-02-22T22:11:39.497 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2026-02-22T22:11:39.497 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:11:39.497 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:11:39.497 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:11:39.497 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:39.497 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:39.498 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:39.498 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T22:11:39.525 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2026-02-22T22:11:39.527 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2026-02-22T22:11:39.527 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:11:39.527 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:11:39.527 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2026-02-22T22:11:39.527 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2026-02-22T22:11:39.528 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:11:39.529 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:11:39.529 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:11:39.529 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:11:39.529 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:39.529 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:39.529 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:11:39.529 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:11:39.529 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get fsid 2026-02-22T22:11:39.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2026-02-22T22:11:39.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2026-02-22T22:11:39.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2026-02-22T22:11:39.584 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2026-02-22T22:11:39.585 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2026-02-22T22:11:39.585 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2026-02-22T22:11:39.585 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:11:39.585 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:11:39.585 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:11:39.586 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:39.586 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:39.586 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:11:39.586 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2026-02-22T22:11:39.586 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.71671/ceph-mon.a.asok config get mon_host 2026-02-22T22:11:39.643 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2026-02-22T22:11:39.643 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2026-02-22T22:11:39.643 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2026-02-22T22:11:39.643 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2026-02-22T22:11:39.643 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2026-02-22T22:11:39.643 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2026-02-22T22:11:39.643 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T22:11:39.752 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2026-02-22T22:11:39.753 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:11:39.753 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:11:39.753 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:11:39.753 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:39.753 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:39.753 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:39.754 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:11:39.755 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T22:11:39.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2026-02-22T22:11:39.773 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2026-02-22T22:11:39.890 INFO:tasks.workunit.client.0.trial034.stderr:pool 'rbd' already exists 2026-02-22T22:11:39.898 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:11:40.900 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2026-02-22T22:11:40.901 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2026-02-22T22:11:40.901 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2026-02-22T22:11:40.901 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:11:40.901 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:40.901 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:40.901 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-mon.a.asok 2026-02-22T22:11:40.901 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2026-02-22T22:11:40.902 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-mon.a.asok log flush 2026-02-22T22:11:40.949 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:11:40.955 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2026-02-22T22:11:40.957 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:11:39.513+0000 7f94fe721ec0 0 load: jerasure load: lrc 2026-02-22T22:11:40.957 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_1 td/test-erasure-eio 2026-02-22T22:11:40.958 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:344: TEST_rados_get_with_subreadall_eio_shard_1: local dir=td/test-erasure-eio 2026-02-22T22:11:40.958 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:345: TEST_rados_get_with_subreadall_eio_shard_1: local shard_id=1 2026-02-22T22:11:40.958 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:347: TEST_rados_get_with_subreadall_eio_shard_1: setup_osds 4 2026-02-22T22:11:40.958 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2026-02-22T22:11:40.958 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2026-02-22T22:11:40.958 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2026-02-22T22:11:40.959 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2026-02-22T22:11:40.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:11:40.962 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2026-02-22T22:11:40.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:11:40.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:11:40.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2026-02-22T22:11:40.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:11:40.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2026-02-22T22:11:40.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:11:40.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:11:40.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:11:40.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:11:40.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2026-02-22T22:11:40.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2026-02-22T22:11:40.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:11:40.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:11:40.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:11:40.964 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:11:40.964 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:11:40.964 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:11:40.964 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:11:40.964 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:40.964 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:40.964 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:40.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:40.964 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:11:40.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:11:40.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:11:40.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:11:40.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:11:40.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:11:40.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:11:40.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:11:40.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:11:40.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:11:40.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:11:40.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2026-02-22T22:11:40.966 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:11:40.966 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6b59c8a1-5de3-4e4f-9fd6-5a92de6a06a3 2026-02-22T22:11:40.966 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6b59c8a1-5de3-4e4f-9fd6-5a92de6a06a3' 2026-02-22T22:11:40.967 INFO:tasks.workunit.client.0.trial034.stdout:add osd0 6b59c8a1-5de3-4e4f-9fd6-5a92de6a06a3 2026-02-22T22:11:40.967 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:11:40.975 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAcf5tpjVswOhAAMjcnhkBPAxQRHguqdUAg2w== 2026-02-22T22:11:40.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAcf5tpjVswOhAAMjcnhkBPAxQRHguqdUAg2w=="}' 2026-02-22T22:11:40.976 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6b59c8a1-5de3-4e4f-9fd6-5a92de6a06a3 -i td/test-erasure-eio/0/new.json 2026-02-22T22:11:41.086 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:11:41.095 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2026-02-22T22:11:41.096 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAcf5tpjVswOhAAMjcnhkBPAxQRHguqdUAg2w== --osd-uuid 6b59c8a1-5de3-4e4f-9fd6-5a92de6a06a3 2026-02-22T22:11:41.112 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:41.109+0000 7fa9bdcb78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:41.113 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:41.109+0000 7fa9bdcb78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:41.115 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:41.109+0000 7fa9bdcb78c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:41.116 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:41.113+0000 7fa9bdcb78c0 -1 bdev(0x563fc07edc00 td/test-erasure-eio/0/block) open stat got: (1) Operation not permitted 2026-02-22T22:11:41.116 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:41.113+0000 7fa9bdcb78c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2026-02-22T22:11:43.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2026-02-22T22:11:43.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:11:43.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2026-02-22T22:11:43.388 INFO:tasks.workunit.client.0.trial034.stdout:adding osd0 key to auth repository 2026-02-22T22:11:43.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:11:43.656 INFO:tasks.workunit.client.0.trial034.stdout:start osd.0 2026-02-22T22:11:43.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2026-02-22T22:11:43.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:11:43.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:11:43.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:11:43.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:11:43.679 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:43.677+0000 7feba93d98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:43.681 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:43.677+0000 7feba93d98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:43.682 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:43.677+0000 7feba93d98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:43.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2026-02-22T22:11:43.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:11:43.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2026-02-22T22:11:43.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:11:43.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:11:43.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:43.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:11:43.856 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:11:43.857 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:43.857 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:11:44.054 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:45.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:45.056 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:45.057 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:11:45.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:11:45.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:45.058 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:11:45.200 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:45.196+0000 7feba93d98c0 -1 Falling back to public interface 2026-02-22T22:11:45.247 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:46.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:46.249 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:46.250 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:11:46.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:11:46.250 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:46.251 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:11:46.442 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:47.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:47.443 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:47.443 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:11:47.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:11:47.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:47.444 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:11:47.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:48.206 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:48.200+0000 7feba93d98c0 -1 osd.0 0 log_to_monitors true 2026-02-22T22:11:48.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:48.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:48.640 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:11:48.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:11:48.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:48.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2026-02-22T22:11:48.839 INFO:tasks.workunit.client.0.trial034.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1504660455,v1:127.0.0.1:6803/1504660455] [v2:127.0.0.1:6804/1504660455,v1:127.0.0.1:6805/1504660455] exists,up 6b59c8a1-5de3-4e4f-9fd6-5a92de6a06a3 2026-02-22T22:11:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:11:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:11:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:11:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:11:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2026-02-22T22:11:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:11:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:11:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2026-02-22T22:11:48.843 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:11:48.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2026-02-22T22:11:48.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:11:48.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:11:48.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:11:48.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:11:48.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2026-02-22T22:11:48.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2026-02-22T22:11:48.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:11:48.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:11:48.844 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:11:48.844 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:11:48.845 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:11:48.845 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:11:48.845 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:11:48.845 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:48.845 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:48.845 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:48.845 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:48.845 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:11:48.845 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:11:48.845 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:11:48.845 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:11:48.845 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:11:48.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:11:48.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:11:48.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:11:48.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:11:48.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:11:48.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:11:48.846 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2026-02-22T22:11:48.846 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:11:48.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6bad8185-1f5f-4427-ba1d-b8adc1e7db04 2026-02-22T22:11:48.847 INFO:tasks.workunit.client.0.trial034.stdout:add osd1 6bad8185-1f5f-4427-ba1d-b8adc1e7db04 2026-02-22T22:11:48.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6bad8185-1f5f-4427-ba1d-b8adc1e7db04' 2026-02-22T22:11:48.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:11:48.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAkf5tpyRQTMxAAVD3bxcZCbF4Evd+Y1Pz/IA== 2026-02-22T22:11:48.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAkf5tpyRQTMxAAVD3bxcZCbF4Evd+Y1Pz/IA=="}' 2026-02-22T22:11:48.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6bad8185-1f5f-4427-ba1d-b8adc1e7db04 -i td/test-erasure-eio/1/new.json 2026-02-22T22:11:49.059 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:11:49.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2026-02-22T22:11:49.069 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkf5tpyRQTMxAAVD3bxcZCbF4Evd+Y1Pz/IA== --osd-uuid 6bad8185-1f5f-4427-ba1d-b8adc1e7db04 2026-02-22T22:11:49.085 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:49.080+0000 7f94d61538c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:49.086 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:49.080+0000 7f94d61538c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:49.087 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:49.084+0000 7f94d61538c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:49.088 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:49.084+0000 7f94d61538c0 -1 bdev(0x55f795709c00 td/test-erasure-eio/1/block) open stat got: (1) Operation not permitted 2026-02-22T22:11:49.088 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:49.084+0000 7f94d61538c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2026-02-22T22:11:51.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2026-02-22T22:11:51.392 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:11:51.393 INFO:tasks.workunit.client.0.trial034.stdout:adding osd1 key to auth repository 2026-02-22T22:11:51.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2026-02-22T22:11:51.393 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:11:51.667 INFO:tasks.workunit.client.0.trial034.stdout:start osd.1 2026-02-22T22:11:51.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2026-02-22T22:11:51.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:11:51.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:11:51.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:11:51.668 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:11:51.690 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:51.684+0000 7fb3b749b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:51.692 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:51.688+0000 7fb3b749b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:51.693 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:51.688+0000 7fb3b749b8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:51.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2026-02-22T22:11:51.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:11:51.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2026-02-22T22:11:51.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:11:51.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:11:51.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:51.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:11:51.870 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:11:51.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:51.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:11:52.065 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:53.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:53.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:53.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:11:53.067 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:11:53.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:53.067 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:11:53.192 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:53.188+0000 7fb3b749b8c0 -1 Falling back to public interface 2026-02-22T22:11:53.259 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:54.260 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:54.261 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:54.261 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:11:54.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:11:54.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:54.262 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:11:54.455 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:55.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:55.456 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:55.457 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:11:55.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:11:55.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:55.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:11:55.654 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:11:56.180 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:56.176+0000 7fb3b749b8c0 -1 osd.1 0 log_to_monitors true 2026-02-22T22:11:56.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:11:56.656 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:56.657 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:11:56.657 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:11:56.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:56.658 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2026-02-22T22:11:56.864 INFO:tasks.workunit.client.0.trial034.stdout:osd.1 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1167563590,v1:127.0.0.1:6811/1167563590] [v2:127.0.0.1:6812/1167563590,v1:127.0.0.1:6813/1167563590] exists,up 6bad8185-1f5f-4427-ba1d-b8adc1e7db04 2026-02-22T22:11:56.867 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:11:56.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:11:56.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:11:56.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:11:56.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2026-02-22T22:11:56.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:11:56.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:11:56.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2026-02-22T22:11:56.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:11:56.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:11:56.868 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:11:56.869 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:11:56.870 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:11:56.870 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:56.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:11:56.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:11:56.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:11:56.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:11:56.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:11:56.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:11:56.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:11:56.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:11:56.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:11:56.870 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:11:56.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:11:56.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:11:56.871 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:11:56.871 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:11:56.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fad4e063-6b2c-4471-8a00-d4a6d930c279 2026-02-22T22:11:56.872 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fad4e063-6b2c-4471-8a00-d4a6d930c279' 2026-02-22T22:11:56.872 INFO:tasks.workunit.client.0.trial034.stdout:add osd2 fad4e063-6b2c-4471-8a00-d4a6d930c279 2026-02-22T22:11:56.872 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:11:56.881 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsf5tpkhiTNBAAJotSqBtEDoxAanDsL2tNXg== 2026-02-22T22:11:56.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsf5tpkhiTNBAAJotSqBtEDoxAanDsL2tNXg=="}' 2026-02-22T22:11:56.882 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fad4e063-6b2c-4471-8a00-d4a6d930c279 -i td/test-erasure-eio/2/new.json 2026-02-22T22:11:57.088 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:11:57.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2026-02-22T22:11:57.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsf5tpkhiTNBAAJotSqBtEDoxAanDsL2tNXg== --osd-uuid fad4e063-6b2c-4471-8a00-d4a6d930c279 2026-02-22T22:11:57.115 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:57.112+0000 7f1b256dc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:57.116 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:57.112+0000 7f1b256dc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:57.117 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:57.112+0000 7f1b256dc8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:57.118 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:57.112+0000 7f1b256dc8c0 -1 bdev(0x55f090d0dc00 td/test-erasure-eio/2/block) open stat got: (1) Operation not permitted 2026-02-22T22:11:57.118 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:57.112+0000 7f1b256dc8c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2026-02-22T22:11:59.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2026-02-22T22:11:59.434 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:11:59.435 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2026-02-22T22:11:59.436 INFO:tasks.workunit.client.0.trial034.stdout:adding osd2 key to auth repository 2026-02-22T22:11:59.436 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:11:59.697 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T22:11:59.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2026-02-22T22:11:59.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:11:59.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:11:59.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:11:59.699 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:11:59.720 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:59.716+0000 7f4e01f0a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:59.722 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:59.716+0000 7f4e01f0a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:59.723 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:11:59.720+0000 7f4e01f0a8c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:11:59.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2026-02-22T22:11:59.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:11:59.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:11:59.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:11:59.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:11:59.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:11:59.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:11:59.900 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:11:59.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:11:59.900 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:12:00.097 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:01.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:01.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:01.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:12:01.098 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:12:01.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:01.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:12:01.204 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:01.200+0000 7f4e01f0a8c0 -1 Falling back to public interface 2026-02-22T22:12:01.296 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:02.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:02.297 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:02.298 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:12:02.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:12:02.298 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:02.299 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:12:02.494 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:03.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:03.496 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:03.496 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:12:03.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:12:03.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:03.497 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:12:03.689 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:04.275 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:04.272+0000 7f4e01f0a8c0 -1 osd.2 0 log_to_monitors true 2026-02-22T22:12:04.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:04.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:04.690 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:12:04.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:12:04.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:04.690 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:12:04.886 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:05.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:05.887 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:05.888 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:12:05.888 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:12:05.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:05.889 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:12:06.080 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1966650777,v1:127.0.0.1:6819/1966650777] [v2:127.0.0.1:6820/1966650777,v1:127.0.0.1:6821/1966650777] exists,up fad4e063-6b2c-4471-8a00-d4a6d930c279 2026-02-22T22:12:06.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:12:06.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:12:06.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:12:06.081 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2026-02-22T22:12:06.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2026-02-22T22:12:06.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2026-02-22T22:12:06.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2026-02-22T22:12:06.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2026-02-22T22:12:06.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2026-02-22T22:12:06.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2026-02-22T22:12:06.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:12:06.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:12:06.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:12:06.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:12:06.082 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2026-02-22T22:12:06.083 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2026-02-22T22:12:06.083 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2026-02-22T22:12:06.083 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2026-02-22T22:12:06.083 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:12:06.085 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2026-02-22T22:12:06.085 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:12:06.086 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:12:06.086 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:12:06.086 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:12:06.086 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:12:06.086 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:12:06.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:12:06.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:12:06.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T22:12:06.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T22:12:06.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:12:06.086 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:12:06.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:12:06.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:12:06.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:12:06.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:12:06.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2026-02-22T22:12:06.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2026-02-22T22:12:06.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2026-02-22T22:12:06.087 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2026-02-22T22:12:06.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd919a7e-c2f4-4a7a-bf03-ef4785424421 2026-02-22T22:12:06.087 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 dd919a7e-c2f4-4a7a-bf03-ef4785424421' 2026-02-22T22:12:06.088 INFO:tasks.workunit.client.0.trial034.stdout:add osd3 dd919a7e-c2f4-4a7a-bf03-ef4785424421 2026-02-22T22:12:06.088 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2026-02-22T22:12:06.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA2f5tp/pTjBRAAte77bPHABKkNnMDDPrtllw== 2026-02-22T22:12:06.098 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA2f5tp/pTjBRAAte77bPHABKkNnMDDPrtllw=="}' 2026-02-22T22:12:06.099 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd919a7e-c2f4-4a7a-bf03-ef4785424421 -i td/test-erasure-eio/3/new.json 2026-02-22T22:12:06.302 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:12:06.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2026-02-22T22:12:06.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2f5tp/pTjBRAAte77bPHABKkNnMDDPrtllw== --osd-uuid dd919a7e-c2f4-4a7a-bf03-ef4785424421 2026-02-22T22:12:06.335 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:06.332+0000 7f19006b98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:12:06.336 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:06.332+0000 7f19006b98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:12:06.337 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:06.332+0000 7f19006b98c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:12:06.338 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:06.332+0000 7f19006b98c0 -1 bdev(0x55ea423abc00 td/test-erasure-eio/3/block) open stat got: (1) Operation not permitted 2026-02-22T22:12:06.338 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:06.332+0000 7f19006b98c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2026-02-22T22:12:08.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2026-02-22T22:12:08.604 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2026-02-22T22:12:08.605 INFO:tasks.workunit.client.0.trial034.stdout:adding osd3 key to auth repository 2026-02-22T22:12:08.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2026-02-22T22:12:08.606 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T22:12:08.873 INFO:tasks.workunit.client.0.trial034.stdout:start osd.3 2026-02-22T22:12:08.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2026-02-22T22:12:08.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:12:08.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2026-02-22T22:12:08.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2026-02-22T22:12:08.874 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2026-02-22T22:12:08.896 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:08.892+0000 7f2b420598c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:12:08.898 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:08.892+0000 7f2b420598c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:12:08.899 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:08.896+0000 7f2b420598c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:12:09.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2026-02-22T22:12:09.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:12:09.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2026-02-22T22:12:09.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:12:09.073 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:12:09.074 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:09.074 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:12:09.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:12:09.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:09.075 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:12:09.269 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:10.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:10.271 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:10.271 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:12:10.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:12:10.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:10.272 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:12:10.408 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:10.404+0000 7f2b420598c0 -1 Falling back to public interface 2026-02-22T22:12:10.465 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:11.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:11.467 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:11.467 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:12:11.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:12:11.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:11.468 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:12:11.659 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:12.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:12.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:12.661 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:12:12.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:12:12.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:12.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:12:12.854 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:13.459 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:13.456+0000 7f2b420598c0 -1 osd.3 0 log_to_monitors true 2026-02-22T22:12:13.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:13.855 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:13.855 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:12:13.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:12:13.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:13.856 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:12:14.059 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:15.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:15.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:15.061 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2026-02-22T22:12:15.062 INFO:tasks.workunit.client.0.trial034.stdout:5 2026-02-22T22:12:15.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:15.062 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2026-02-22T22:12:15.263 INFO:tasks.workunit.client.0.trial034.stdout:osd.3 up in weight 1 up_from 24 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2786930122,v1:127.0.0.1:6827/2786930122] [v2:127.0.0.1:6828/2786930122,v1:127.0.0.1:6829/2786930122] exists,up dd919a7e-c2f4-4a7a-bf03-ef4785424421 2026-02-22T22:12:15.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:12:15.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:12:15.265 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:12:15.265 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2026-02-22T22:12:15.265 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2026-02-22T22:12:15.265 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2026-02-22T22:12:15.265 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:12:15.266 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:12:15.266 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:12:15.266 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.0.asok 2026-02-22T22:12:15.266 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2026-02-22T22:12:15.267 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.0.asok log flush 2026-02-22T22:12:15.326 INFO:tasks.workunit.client.0.trial034.stdout:{} 2026-02-22T22:12:15.332 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2026-02-22T22:12:15.334 INFO:tasks.workunit.client.0.trial034.stdout:2026-02-22T22:11:45.200+0000 7feba93d98c0 0 load: jerasure load: lrc 2026-02-22T22:12:15.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:349: TEST_rados_get_with_subreadall_eio_shard_1: local poolname=pool-jerasure 2026-02-22T22:12:15.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:350: TEST_rados_get_with_subreadall_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2026-02-22T22:12:15.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:12:15.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2026-02-22T22:12:15.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2026-02-22T22:12:15.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2026-02-22T22:12:15.335 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2026-02-22T22:12:15.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2026-02-22T22:12:15.336 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2026-02-22T22:12:15.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2026-02-22T22:12:15.588 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2026-02-22T22:12:15.937 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' already exists 2026-02-22T22:12:15.946 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2026-02-22T22:12:16.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2026-02-22T22:12:16.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:12:16.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:12:16.948 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:12:16.949 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:12:16.949 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:12:16.949 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:12:16.950 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:12:16.950 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:12:16.950 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:12:17.059 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:12:17.059 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:12:17.060 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:12:17.060 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:12:17.060 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:12:17.060 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:12:17.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:12:17.258 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:12:17.258 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:12:17.258 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:12:17.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:12:17.258 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:12:17.259 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:12:17.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803784 2026-02-22T22:12:17.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803784 2026-02-22T22:12:17.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784' 2026-02-22T22:12:17.333 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:12:17.333 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:12:17.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=47244640263 2026-02-22T22:12:17.409 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 47244640263 2026-02-22T22:12:17.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-47244640263' 2026-02-22T22:12:17.410 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:12:17.410 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:12:17.484 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=73014444037 2026-02-22T22:12:17.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 73014444037 2026-02-22T22:12:17.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-47244640263 2-73014444037' 2026-02-22T22:12:17.485 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:12:17.485 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:12:17.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=103079215107 2026-02-22T22:12:17.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 103079215107 2026-02-22T22:12:17.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803784 1-47244640263 2-73014444037 3-103079215107' 2026-02-22T22:12:17.560 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:12:17.560 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803784 2026-02-22T22:12:17.561 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:12:17.561 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:12:17.562 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803784 2026-02-22T22:12:17.562 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:12:17.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803784 2026-02-22T22:12:17.564 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2026-02-22T22:12:17.564 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803784 2026-02-22T22:12:17.564 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:12:17.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803782 -lt 25769803784 2026-02-22T22:12:17.757 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:12:18.758 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:12:18.759 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:12:18.961 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803784 -lt 25769803784 2026-02-22T22:12:18.961 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:12:18.962 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-47244640263 2026-02-22T22:12:18.962 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:12:18.963 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:12:18.964 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-47244640263 2026-02-22T22:12:18.964 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:12:18.965 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=47244640263 2026-02-22T22:12:18.966 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 47244640263 2026-02-22T22:12:18.966 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 47244640263' 2026-02-22T22:12:18.966 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:12:19.170 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 47244640263 -lt 47244640263 2026-02-22T22:12:19.170 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:12:19.171 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-73014444037 2026-02-22T22:12:19.171 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:12:19.172 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:12:19.172 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-73014444037 2026-02-22T22:12:19.173 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:12:19.174 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=73014444037 2026-02-22T22:12:19.174 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 73014444037 2026-02-22T22:12:19.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 73014444037' 2026-02-22T22:12:19.175 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:12:19.386 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 73014444037 -lt 73014444037 2026-02-22T22:12:19.387 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:12:19.387 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-103079215107 2026-02-22T22:12:19.387 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:12:19.388 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:12:19.389 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-103079215107 2026-02-22T22:12:19.389 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:12:19.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=103079215107 2026-02-22T22:12:19.391 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 103079215107' 2026-02-22T22:12:19.391 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 103079215107 2026-02-22T22:12:19.392 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:12:19.589 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 103079215107 -lt 103079215107 2026-02-22T22:12:19.589 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:12:19.590 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:12:19.590 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:12:19.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:12:19.842 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:12:19.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:12:19.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:12:19.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:12:19.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:12:19.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:12:19.843 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:12:20.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:12:20.035 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:12:20.036 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:12:20.036 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:12:20.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:12:20.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:12:20.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:12:20.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:352: TEST_rados_get_with_subreadall_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 recovery 2026-02-22T22:12:20.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2026-02-22T22:12:20.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2026-02-22T22:12:20.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2026-02-22T22:12:20.288 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-71671-1 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-71671-1 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-71671-1 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2026-02-22T22:12:20.289 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-71671-1 td/test-erasure-eio/ORIGINAL 2026-02-22T22:12:20.310 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-71671-1 td/test-erasure-eio 1 2026-02-22T22:12:20.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: inject_eio: local pooltype=ec 2026-02-22T22:12:20.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: inject_eio: shift 2026-02-22T22:12:20.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: inject_eio: local which=data 2026-02-22T22:12:20.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: inject_eio: shift 2026-02-22T22:12:20.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: inject_eio: local poolname=pool-jerasure 2026-02-22T22:12:20.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: inject_eio: shift 2026-02-22T22:12:20.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2463: inject_eio: local objname=obj-eio-71671-1 2026-02-22T22:12:20.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: inject_eio: shift 2026-02-22T22:12:20.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: inject_eio: local dir=td/test-erasure-eio 2026-02-22T22:12:20.311 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: inject_eio: shift 2026-02-22T22:12:20.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2467: inject_eio: local shard_id=1 2026-02-22T22:12:20.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2468: inject_eio: shift 2026-02-22T22:12:20.312 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: get_osds pool-jerasure obj-eio-71671-1 2026-02-22T22:12:20.312 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:12:20.312 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-71671-1 2026-02-22T22:12:20.312 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-71671-1 2026-02-22T22:12:20.313 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:12:20.519 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:12:20.519 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:12:20.519 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:12:20.519 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:12:20.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: initial_osds=('3' '1' '2') 2026-02-22T22:12:20.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2470: inject_eio: local -a initial_osds 2026-02-22T22:12:20.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2471: inject_eio: local osd_id=1 2026-02-22T22:12:20.520 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2472: inject_eio: '[' ec '!=' ec ']' 2026-02-22T22:12:20.520 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: cat td/test-erasure-eio/1/type 2026-02-22T22:12:20.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2475: inject_eio: type=bluestore 2026-02-22T22:12:20.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2476: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-22T22:12:20.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2026-02-22T22:12:20.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2026-02-22T22:12:20.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T22:12:20.521 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2026-02-22T22:12:20.521 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2026-02-22T22:12:20.522 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2026-02-22T22:12:20.522 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2026-02-22T22:12:20.522 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2026-02-22T22:12:20.522 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:12:20.522 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:12:20.523 INFO:tasks.workunit.client.0.trial034.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:12:20.523 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.1.asok 2026-02-22T22:12:20.523 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.71671/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-22T22:12:20.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2026-02-22T22:12:20.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2477: inject_eio: local loop=0 2026-02-22T22:12:20.582 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2479: inject_eio: grep -q Invalid 2026-02-22T22:12:20.583 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: get_asok_path osd.1 2026-02-22T22:12:20.583 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2026-02-22T22:12:20.583 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2026-02-22T22:12:20.583 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2026-02-22T22:12:20.583 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:12:20.583 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:12:20.583 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71671/ceph-osd.1.asok 2026-02-22T22:12:20.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: CEPH_ARGS= 2026-02-22T22:12:20.584 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2478: inject_eio: ceph --admin-daemon /tmp/ceph-asok.71671/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-71671-1 1 2026-02-22T22:12:20.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-71671-1 2026-02-22T22:12:20.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2026-02-22T22:12:20.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2026-02-22T22:12:20.640 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-71671-1 2026-02-22T22:12:20.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2026-02-22T22:12:20.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2026-02-22T22:12:20.641 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-71671-1 td/test-erasure-eio/COPY 2026-02-22T22:12:20.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2026-02-22T22:12:20.661 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2026-02-22T22:12:20.662 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2026-02-22T22:12:20.663 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-71671-1 2026-02-22T22:12:20.663 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:12:20.663 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-71671-1 2026-02-22T22:12:20.664 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-71671-1 2026-02-22T22:12:20.664 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:12:20.861 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:12:20.861 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:12:20.861 INFO:tasks.workunit.client.0.trial034.stderr:2' 2026-02-22T22:12:20.861 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2026-02-22T22:12:20.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2026-02-22T22:12:20.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2026-02-22T22:12:20.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2026-02-22T22:12:20.861 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2026-02-22T22:12:20.861 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:12:20.862 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:12:20.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:12:20.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:12:20.862 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:12:22.171 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:12:22.172 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2026-02-22T22:12:22.418 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 is already out. 2026-02-22T22:12:22.428 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-71671-1 2026-02-22T22:12:22.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2026-02-22T22:12:22.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-71671-1 2026-02-22T22:12:22.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2026-02-22T22:12:22.429 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-71671-1 2026-02-22T22:12:22.429 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2026-02-22T22:12:22.630 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2026-02-22T22:12:22.630 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:12:22.630 INFO:tasks.workunit.client.0.trial034.stderr:0' 2026-02-22T22:12:22.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2026-02-22T22:12:22.631 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2026-02-22T22:12:22.822 INFO:tasks.workunit.client.0.trial034.stderr:osd.2 is already in. 2026-02-22T22:12:22.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2026-02-22T22:12:22.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2026-02-22T22:12:22.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2026-02-22T22:12:22.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2026-02-22T22:12:22.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2026-02-22T22:12:22.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2026-02-22T22:12:22.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2026-02-22T22:12:22.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T22:12:22.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T22:12:22.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T22:12:22.831 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2026-02-22T22:12:22.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2026-02-22T22:12:22.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2026-02-22T22:12:22.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2026-02-22T22:12:22.832 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2026-02-22T22:12:22.832 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2026-02-22T22:12:22.832 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2026-02-22T22:12:22.832 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2026-02-22T22:12:22.832 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2026-02-22T22:12:22.832 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:12:22.832 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:12:22.832 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:12:22.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' 2026-02-22T22:12:22.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T22:12:22.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2026-02-22T22:12:22.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2026-02-22T22:12:22.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T22:12:22.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T22:12:22.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T22:12:22.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T22:12:22.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2026-02-22T22:12:22.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2026-02-22T22:12:22.833 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2026-02-22T22:12:22.834 INFO:tasks.workunit.client.0.trial034.stdout:start osd.2 2026-02-22T22:12:22.834 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2026-02-22T22:12:22.834 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=224e6b9a-0ae0-441d-a396-63aa6115a4f9 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.71671/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T22:12:22.834 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2026-02-22T22:12:22.835 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2026-02-22T22:12:22.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2026-02-22T22:12:22.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2026-02-22T22:12:22.836 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2026-02-22T22:12:22.858 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:22.856+0000 7f7775e588c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:12:22.859 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:22.860+0000 7f7775e588c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:12:22.860 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:22.860+0000 7f7775e588c0 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T22:12:23.034 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2026-02-22T22:12:23.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2026-02-22T22:12:23.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2026-02-22T22:12:23.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2026-02-22T22:12:23.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2026-02-22T22:12:23.035 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:23.035 INFO:tasks.workunit.client.0.trial034.stdout:0 2026-02-22T22:12:23.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2026-02-22T22:12:23.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:23.036 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:12:23.227 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:24.169 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:24.168+0000 7f7775e588c0 -1 Falling back to public interface 2026-02-22T22:12:24.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:24.229 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:24.229 INFO:tasks.workunit.client.0.trial034.stdout:1 2026-02-22T22:12:24.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2026-02-22T22:12:24.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:24.230 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:12:24.422 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:25.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:25.424 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:25.424 INFO:tasks.workunit.client.0.trial034.stdout:2 2026-02-22T22:12:25.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2026-02-22T22:12:25.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:25.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:12:25.617 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:26.618 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:26.619 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:26.619 INFO:tasks.workunit.client.0.trial034.stdout:3 2026-02-22T22:12:26.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2026-02-22T22:12:26.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:26.620 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:12:26.812 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2026-02-22T22:12:27.209 INFO:tasks.workunit.client.0.trial034.stderr:2026-02-22T22:12:27.208+0000 7f7775e588c0 -1 osd.2 32 log_to_monitors true 2026-02-22T22:12:27.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2026-02-22T22:12:27.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2026-02-22T22:12:27.813 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2026-02-22T22:12:27.814 INFO:tasks.workunit.client.0.trial034.stdout:4 2026-02-22T22:12:27.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2026-02-22T22:12:27.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2026-02-22T22:12:28.004 INFO:tasks.workunit.client.0.trial034.stdout:osd.2 up in weight 1 up_from 41 up_thru 17 down_at 33 last_clean_interval [17,32) [v2:127.0.0.1:6818/1646858076,v1:127.0.0.1:6819/1646858076] [v2:127.0.0.1:6820/1646858076,v1:127.0.0.1:6821/1646858076] exists,up fad4e063-6b2c-4471-8a00-d4a6d930c279 2026-02-22T22:12:28.007 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2026-02-22T22:12:28.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2026-02-22T22:12:28.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2026-02-22T22:12:28.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2026-02-22T22:12:28.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: local cmd= 2026-02-22T22:12:28.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: local num_active_clean=-1 2026-02-22T22:12:28.008 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: local cur_active_clean 2026-02-22T22:12:28.008 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T22:12:28.009 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: shopt -q -o xtrace 2026-02-22T22:12:28.009 INFO:tasks.workunit.client.0.trial034.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: echo true 2026-02-22T22:12:28.009 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: get_timeout_delays: local trace=true 2026-02-22T22:12:28.009 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: true 2026-02-22T22:12:28.009 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: get_timeout_delays: shopt -u -o xtrace 2026-02-22T22:12:28.111 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T22:12:28.111 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: local -a delays 2026-02-22T22:12:28.111 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: local -i loop=0 2026-02-22T22:12:28.112 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: flush_pg_stats 2026-02-22T22:12:28.112 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: local timeout=300 2026-02-22T22:12:28.112 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ceph osd ls 2026-02-22T22:12:28.306 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: ids='0 2026-02-22T22:12:28.307 INFO:tasks.workunit.client.0.trial034.stderr:1 2026-02-22T22:12:28.307 INFO:tasks.workunit.client.0.trial034.stderr:2 2026-02-22T22:12:28.307 INFO:tasks.workunit.client.0.trial034.stderr:3' 2026-02-22T22:12:28.307 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs= 2026-02-22T22:12:28.307 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:12:28.307 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T22:12:28.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=25769803788 2026-02-22T22:12:28.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 25769803788 2026-02-22T22:12:28.379 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803788' 2026-02-22T22:12:28.380 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:12:28.380 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T22:12:28.457 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=47244640267 2026-02-22T22:12:28.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 47244640267 2026-02-22T22:12:28.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803788 1-47244640267' 2026-02-22T22:12:28.458 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:12:28.458 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T22:12:28.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=176093659139 2026-02-22T22:12:28.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 176093659139 2026-02-22T22:12:28.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803788 1-47244640267 2-176093659139' 2026-02-22T22:12:28.532 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: flush_pg_stats: for osd in $ids 2026-02-22T22:12:28.532 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T22:12:28.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: seq=103079215111 2026-02-22T22:12:28.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: test -z 103079215111 2026-02-22T22:12:28.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: seqs=' 0-25769803788 1-47244640267 2-176093659139 3-103079215111' 2026-02-22T22:12:28.608 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:12:28.609 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 0-25769803788 2026-02-22T22:12:28.609 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:12:28.610 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=0 2026-02-22T22:12:28.610 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 0-25769803788 2026-02-22T22:12:28.610 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:12:28.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=25769803788 2026-02-22T22:12:28.612 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.0 seq 25769803788' 2026-02-22T22:12:28.612 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.0 seq 25769803788 2026-02-22T22:12:28.613 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:12:28.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803786 -lt 25769803788 2026-02-22T22:12:28.814 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:12:29.815 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T22:12:29.815 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:12:30.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803786 -lt 25769803788 2026-02-22T22:12:30.021 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: flush_pg_stats: sleep 1 2026-02-22T22:12:31.022 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T22:12:31.023 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T22:12:31.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 25769803789 -lt 25769803788 2026-02-22T22:12:31.216 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:12:31.217 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 1-47244640267 2026-02-22T22:12:31.217 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:12:31.218 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=1 2026-02-22T22:12:31.218 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 1-47244640267 2026-02-22T22:12:31.219 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:12:31.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=47244640267 2026-02-22T22:12:31.220 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.1 seq 47244640267 2026-02-22T22:12:31.220 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.1 seq 47244640267' 2026-02-22T22:12:31.221 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T22:12:31.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 47244640267 -lt 47244640267 2026-02-22T22:12:31.425 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:12:31.425 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 2-176093659139 2026-02-22T22:12:31.425 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:12:31.427 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=2 2026-02-22T22:12:31.428 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 2-176093659139 2026-02-22T22:12:31.428 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:12:31.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=176093659139 2026-02-22T22:12:31.429 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.2 seq 176093659139 2026-02-22T22:12:31.429 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.2 seq 176093659139' 2026-02-22T22:12:31.430 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T22:12:31.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 176093659139 -lt 176093659139 2026-02-22T22:12:31.634 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: flush_pg_stats: for s in $seqs 2026-02-22T22:12:31.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: echo 3-103079215111 2026-02-22T22:12:31.635 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: cut -d - -f 1 2026-02-22T22:12:31.636 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: flush_pg_stats: osd=3 2026-02-22T22:12:31.636 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: echo 3-103079215111 2026-02-22T22:12:31.636 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: cut -d - -f 2 2026-02-22T22:12:31.638 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: flush_pg_stats: seq=103079215111 2026-02-22T22:12:31.638 INFO:tasks.workunit.client.0.trial034.stdout:waiting osd.3 seq 103079215111 2026-02-22T22:12:31.639 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: flush_pg_stats: echo 'waiting osd.3 seq 103079215111' 2026-02-22T22:12:31.639 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T22:12:31.847 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: flush_pg_stats: test 103079215112 -lt 103079215111 2026-02-22T22:12:31.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T22:12:31.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:12:31.848 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:12:32.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 == 0 2026-02-22T22:12:32.107 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: true 2026-02-22T22:12:32.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: get_num_active_clean 2026-02-22T22:12:32.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2026-02-22T22:12:32.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T22:12:32.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T22:12:32.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T22:12:32.107 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T22:12:32.312 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: cur_active_clean=5 2026-02-22T22:12:32.312 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: get_num_pgs 2026-02-22T22:12:32.312 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2026-02-22T22:12:32.312 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T22:12:32.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: test 5 = 5 2026-02-22T22:12:32.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1672: wait_for_clean: break 2026-02-22T22:12:32.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: wait_for_clean: return 0 2026-02-22T22:12:32.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:354: TEST_rados_get_with_subreadall_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2026-02-22T22:12:32.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2026-02-22T22:12:32.577 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2026-02-22T22:12:32.785 INFO:tasks.workunit.client.0.trial034.stderr:pool 'pool-jerasure' does not exist 2026-02-22T22:12:32.794 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2026-02-22T22:12:33.055 INFO:tasks.workunit.client.0.trial034.stderr:erasure-code-profile myprofile does not exist 2026-02-22T22:12:33.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2026-02-22T22:12:33.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:12:33.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2026-02-22T22:12:33.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:12:33.064 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:12:33.064 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:12:33.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:12:33.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:12:33.064 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:12:33.175 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:12:33.175 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:12:33.176 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:12:33.177 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:12:33.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:12:33.178 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:12:33.179 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:12:33.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:12:33.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:12:33.180 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:12:33.181 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:12:33.182 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:12:33.183 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T22:12:33.183 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:12:33.199 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:12:33.199 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:12:33.199 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:12:33.199 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:12:33.200 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:12:33.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:12:33.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2377: main: code=0 2026-02-22T22:12:33.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: main: teardown td/test-erasure-eio 0 2026-02-22T22:12:33.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2026-02-22T22:12:33.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2026-02-22T22:12:33.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2026-02-22T22:12:33.201 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2026-02-22T22:12:33.201 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2026-02-22T22:12:33.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2026-02-22T22:12:33.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2026-02-22T22:12:33.201 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2026-02-22T22:12:33.204 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2026-02-22T22:12:33.204 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2026-02-22T22:12:33.205 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T22:12:33.206 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2026-02-22T22:12:33.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T22:12:33.207 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2026-02-22T22:12:33.207 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2026-02-22T22:12:33.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:12:33.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2026-02-22T22:12:33.209 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2026-02-22T22:12:33.209 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T22:12:33.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:12:33.211 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T22:12:33.212 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2026-02-22T22:12:33.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2026-02-22T22:12:33.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2026-02-22T22:12:33.213 INFO:tasks.workunit.client.0.trial034.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71671 2026-02-22T22:12:33.213 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71671 2026-02-22T22:12:33.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2026-02-22T22:12:33.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2026-02-22T22:12:33.215 INFO:tasks.workunit.client.0.trial034.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2382: main: return 0 2026-02-22T22:12:33.216 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T22:12:33.216 DEBUG:teuthology.orchestra.run.trial034:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T22:12:33.266 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2026-02-22T22:12:33.266 DEBUG:teuthology.orchestra.run.trial034:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-22T22:12:33.611 DEBUG:teuthology.parallel:result is None 2026-02-22T22:12:33.611 DEBUG:teuthology.orchestra.run.trial034:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-22T22:12:33.618 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-22T22:12:33.618 DEBUG:teuthology.orchestra.run.trial034:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-22T22:12:33.662 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-22T22:12:33.662 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-22T22:12:33.675 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-22T22:12:33.675 DEBUG:teuthology.orchestra.run.trial034:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T22:12:33.814 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-22T22:12:33.825 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-22T22:12:33.825 DEBUG:teuthology.orchestra.run.trial034:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T22:12:34.051 INFO:teuthology.orchestra.run.trial034.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T22:12:34.051 INFO:teuthology.orchestra.run.trial034.stdout:============================================================================== 2026-02-22T22:12:34.052 INFO:teuthology.orchestra.run.trial034.stdout:-204.2.134.173 44.24.199.34 3 u 127 64 376 68.233 -1.217 1.905 2026-02-22T22:12:34.052 INFO:teuthology.orchestra.run.trial034.stdout:*108.61.73.244 129.6.15.27 2 u 52 64 377 3.426 -2.269 0.429 2026-02-22T22:12:34.052 INFO:teuthology.orchestra.run.trial034.stdout:+23.186.168.132 132.163.97.4 2 u 59 64 377 69.694 -3.332 5.173 2026-02-22T22:12:34.052 INFO:teuthology.orchestra.run.trial034.stdout:+172-233-155-39. 40.119.6.228 4 u 55 64 377 66.497 -1.288 2.164 2026-02-22T22:12:34.053 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-22T22:12:34.067 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-22T22:12:34.067 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-22T22:12:34.080 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-22T22:12:34.095 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-22T22:12:34.110 INFO:teuthology.task.internal:Duration was 1719.643985 seconds 2026-02-22T22:12:34.111 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-22T22:12:34.126 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-22T22:12:34.126 DEBUG:teuthology.orchestra.run.trial034:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T22:12:34.152 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-22T22:12:34.153 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial034.front.sepia.ceph.com 2026-02-22T22:12:34.153 DEBUG:teuthology.orchestra.run.trial034:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T22:12:34.200 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-22T22:12:34.201 DEBUG:teuthology.orchestra.run.trial034:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T22:12:34.461 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-22T22:12:34.461 DEBUG:teuthology.orchestra.run.trial034:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T22:12:34.471 INFO:teuthology.orchestra.run.trial034.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T22:12:34.471 INFO:teuthology.orchestra.run.trial034.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T22:12:34.472 INFO:teuthology.orchestra.run.trial034.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T22:12:34.472 INFO:teuthology.orchestra.run.trial034.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T22:12:34.472 INFO:teuthology.orchestra.run.trial034.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T22:12:34.539 INFO:teuthology.orchestra.run.trial034.stderr: 81.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T22:12:34.541 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-22T22:12:34.557 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-22T22:12:34.558 DEBUG:teuthology.orchestra.run.trial034:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T22:12:34.591 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-22T22:12:34.607 DEBUG:teuthology.orchestra.run.trial034:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:12:34.642 INFO:teuthology.orchestra.run.trial034.stdout:kernel.core_pattern = core 2026-02-22T22:12:34.652 DEBUG:teuthology.orchestra.run.trial034:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T22:12:34.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T22:12:34.698 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-22T22:12:34.716 INFO:teuthology.task.internal:Transferring archived files... 2026-02-22T22:12:34.718 DEBUG:teuthology.misc:Transferring archived files from trial034:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-22_19:35:40-rados-wip-yuri6-testing-2026-02-22-1629-squid-distro-default-trial/64109/remote/trial034 2026-02-22T22:12:34.720 DEBUG:teuthology.orchestra.run.trial034:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T22:12:34.782 INFO:teuthology.task.internal:Removing archive directory... 2026-02-22T22:12:34.782 DEBUG:teuthology.orchestra.run.trial034:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T22:12:34.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-22T22:12:34.824 INFO:teuthology.task.internal:Not uploading archives. 2026-02-22T22:12:34.824 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-22T22:12:34.844 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-22T22:12:34.844 DEBUG:teuthology.orchestra.run.trial034:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T22:12:34.849 INFO:teuthology.orchestra.run.trial034.stdout: 1179687 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 22:12 /home/ubuntu/cephtest 2026-02-22T22:12:34.850 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-22T22:12:34.862 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-22T22:12:34.887 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} duration: 1719.6439845561981 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-22T22:12:34.887 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T22:12:34.996 INFO:teuthology.run:pass